2024-04-27T03:44:10.152 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-27T03:44:10.152 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --name teuthology-2024-04-27_03:08:09-upgrade-main-distro-default-smithi --verbose --description upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --archive /home/teuthworker/archive/teuthology-2024-04-27_03:08:09-upgrade-main-distro-default-smithi/7675777 -- /home/teuthworker/archive/teuthology-2024-04-27_03:08:09-upgrade-main-distro-default-smithi/7675777/orig.config.yaml 2024-04-27T03:44:10.205 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-27T03:44:10.295 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-27_03:08:09-upgrade-main-distro-default-smithi/7675777 branch: main description: upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7675777' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/reef ' name: teuthology-2024-04-27_03:08:09-upgrade-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 4e213a58e787437ff508e6c890ccd5fd32dc96b5 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 4e213a58e787437ff508e6c890ccd5fd32dc96b5 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 3067e9f133f954b642fdf21114a79edb58590948 owner: scheduled_teuthology@teuthology priority: 850 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 1487 sha1: 4e213a58e787437ff508e6c890ccd5fd32dc96b5 sleep_before_teardown: 0 subset: 16/32 suite: upgrade suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 3067e9f133f954b642fdf21114a79edb58590948 targets: smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN8oH4piT6dsshEaKEHjPhB9qziQxQLIzhIPaH6npcdj5tUJpND+yC26uLg+lUe8/WUJwV+YV1lDUHFKbAfbRG8= smithi073.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPOUmIHJ1G/Dby/RkB4+wbyo3AYAEeb02AiGYv/PoYzQzexIeTL4B4X8pqVMHLUxJNDWSKdCHC3xutGxKrdGNpk= tasks: - install: branch: reef exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:reef roleless: true - print: '**** done end installing reef cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /home/teuthology/src/git.ceph.com_ceph_main/qa/suites/upgrade/cephfs/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-27_03:08:09 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-04-27T03:44:10.296 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa; will attempt to use it 2024-04-27T03:44:10.296 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks 2024-04-27T03:44:10.297 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-27T03:44:10.298 INFO:teuthology.task.internal:Checking packages... 2024-04-27T03:44:10.323 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '4e213a58e787437ff508e6c890ccd5fd32dc96b5' 2024-04-27T03:44:10.323 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-27T03:44:10.323 INFO:teuthology.packaging:ref: None 2024-04-27T03:44:10.323 INFO:teuthology.packaging:tag: None 2024-04-27T03:44:10.323 INFO:teuthology.packaging:branch: main 2024-04-27T03:44:10.323 INFO:teuthology.packaging:sha1: 4e213a58e787437ff508e6c890ccd5fd32dc96b5 2024-04-27T03:44:10.323 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-04-27T03:44:10.557 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1670.g1fb5d106 2024-04-27T03:44:10.559 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-27T03:44:10.565 INFO:teuthology.task.internal:no buildpackages task found 2024-04-27T03:44:10.565 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-27T03:44:10.587 INFO:teuthology.task.internal:Saving configuration 2024-04-27T03:44:10.607 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-27T03:44:10.613 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T03:44:10.639 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-27_03:08:09-upgrade-main-distro-default-smithi/7675777', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 03:39:02.274705', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN8oH4piT6dsshEaKEHjPhB9qziQxQLIzhIPaH6npcdj5tUJpND+yC26uLg+lUe8/WUJwV+YV1lDUHFKbAfbRG8='} 2024-04-27T03:44:10.661 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi073.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-27_03:08:09-upgrade-main-distro-default-smithi/7675777', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 03:39:02.276077', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPOUmIHJ1G/Dby/RkB4+wbyo3AYAEeb02AiGYv/PoYzQzexIeTL4B4X8pqVMHLUxJNDWSKdCHC3xutGxKrdGNpk='} 2024-04-27T03:44:10.662 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-27T03:44:10.720 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-27T03:44:10.721 INFO:teuthology.task.internal:roles: ubuntu@smithi073.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-27T03:44:10.721 INFO:teuthology.run_tasks:Running task console_log... 2024-04-27T03:44:10.787 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6e8a9b20d0>, signals=[15]) 2024-04-27T03:44:10.787 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-27T03:44:10.794 INFO:teuthology.task.internal:Opening connections... 2024-04-27T03:44:10.794 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2024-04-27T03:44:10.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T03:44:10.873 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi073.front.sepia.ceph.com 2024-04-27T03:44:10.874 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T03:44:10.951 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-27T03:44:10.958 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2024-04-27T03:44:10.976 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2024-04-27T03:44:10.976 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2024-04-27T03:44:11.035 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Stream" 2024-04-27T03:44:11.035 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="9" 2024-04-27T03:44:11.035 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos" 2024-04-27T03:44:11.035 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora" 2024-04-27T03:44:11.035 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="9" 2024-04-27T03:44:11.035 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el9" 2024-04-27T03:44:11.036 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-27T03:44:11.036 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2024-04-27T03:44:11.036 INFO:teuthology.orchestra.run.smithi046.stdout:LOGO="fedora-logo-icon" 2024-04-27T03:44:11.036 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-27T03:44:11.036 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://centos.org/" 2024-04-27T03:44:11.036 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-27T03:44:11.036 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-27T03:44:11.036 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-27T03:44:11.037 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2024-04-27T03:44:11.065 DEBUG:teuthology.orchestra.run.smithi073:> uname -m 2024-04-27T03:44:11.083 INFO:teuthology.orchestra.run.smithi073.stdout:x86_64 2024-04-27T03:44:11.083 DEBUG:teuthology.orchestra.run.smithi073:> cat /etc/os-release 2024-04-27T03:44:11.139 INFO:teuthology.orchestra.run.smithi073.stdout:NAME="CentOS Stream" 2024-04-27T03:44:11.139 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION="9" 2024-04-27T03:44:11.139 INFO:teuthology.orchestra.run.smithi073.stdout:ID="centos" 2024-04-27T03:44:11.139 INFO:teuthology.orchestra.run.smithi073.stdout:ID_LIKE="rhel fedora" 2024-04-27T03:44:11.139 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_ID="9" 2024-04-27T03:44:11.139 INFO:teuthology.orchestra.run.smithi073.stdout:PLATFORM_ID="platform:el9" 2024-04-27T03:44:11.139 INFO:teuthology.orchestra.run.smithi073.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-27T03:44:11.140 INFO:teuthology.orchestra.run.smithi073.stdout:ANSI_COLOR="0;31" 2024-04-27T03:44:11.140 INFO:teuthology.orchestra.run.smithi073.stdout:LOGO="fedora-logo-icon" 2024-04-27T03:44:11.140 INFO:teuthology.orchestra.run.smithi073.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-27T03:44:11.140 INFO:teuthology.orchestra.run.smithi073.stdout:HOME_URL="https://centos.org/" 2024-04-27T03:44:11.140 INFO:teuthology.orchestra.run.smithi073.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-27T03:44:11.140 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-27T03:44:11.140 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-27T03:44:11.140 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2024-04-27T03:44:11.167 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-27T03:44:11.176 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-27T03:44:11.182 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-27T03:44:11.182 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2024-04-27T03:44:11.185 DEBUG:teuthology.orchestra.run.smithi073:> test '!' -e /home/ubuntu/cephtest 2024-04-27T03:44:11.200 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-27T03:44:11.206 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-27T03:44:11.206 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2024-04-27T03:44:11.242 DEBUG:teuthology.orchestra.run.smithi073:> test -z $(ls -A /var/lib/ceph) 2024-04-27T03:44:11.487 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-27T03:44:11.539 INFO:teuthology.run_tasks:Running task kernel... 2024-04-27T03:44:11.556 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-27T03:44:11.556 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-27T03:44:11.557 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-27T03:44:11.557 DEBUG:teuthology.orchestra.run.smithi073:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-27T03:44:11.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T03:44:11.574 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2024-04-27T03:44:11.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T03:44:11.575 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-04-27T03:44:11.630 INFO:teuthology.orchestra.run.smithi073.stdout:5.14.0-437.el9.x86_64 2024-04-27T03:44:11.630 INFO:teuthology.task.kernel:Running kernel on smithi073: 5.14.0-437.el9.x86_64 2024-04-27T03:44:11.631 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum install -y kernel 2024-04-27T03:44:11.632 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-437.el9.x86_64 2024-04-27T03:44:11.633 INFO:teuthology.task.kernel:Running kernel on smithi046: 5.14.0-437.el9.x86_64 2024-04-27T03:44:11.633 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum install -y kernel 2024-04-27T03:44:14.888 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.0 MB 00:01 2024-04-27T03:44:15.202 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.0 MB 00:01 2024-04-27T03:44:17.053 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-04-27T03:44:21.429 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.2 MB 00:00 2024-04-27T03:44:22.980 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-04-27T03:44:37.244 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - AppStream 915 kB/s | 19 MB 00:21 2024-04-27T03:44:41.544 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.2 MB 00:00 2024-04-27T03:44:43.161 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-04-27T03:44:44.855 INFO:teuthology.orchestra.run.smithi073.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-27T03:44:49.946 INFO:teuthology.orchestra.run.smithi073.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-27T03:44:51.802 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-27T03:44:51.803 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-27T03:44:51.899 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:44:51.900 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-04-27T03:44:51.900 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:44:52.018 DEBUG:teuthology.orchestra.run.smithi073:> echo no | sudo yum reinstall kernel || true 2024-04-27T03:44:52.513 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:03 ago on Sat 27 Apr 2024 03:44:49 AM UTC. 2024-04-27T03:44:52.644 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:44:52.660 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:44:52.660 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2024-04-27T03:44:52.660 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:44:52.660 INFO:teuthology.orchestra.run.smithi073.stdout:Reinstalling: 2024-04-27T03:44:52.660 INFO:teuthology.orchestra.run.smithi073.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T03:44:52.660 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:44:52.660 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:44:52.660 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:44:52.660 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:44:52.661 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 255 k 2024-04-27T03:44:52.661 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 0 2024-04-27T03:44:52.661 INFO:teuthology.orchestra.run.smithi073.stderr:Operation aborted. 2024-04-27T03:44:52.720 INFO:teuthology.orchestra.run.smithi073.stdout:Is this ok [y/N]: 2024-04-27T03:44:52.720 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum reinstall -y kernel || true 2024-04-27T03:44:53.211 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:04 ago on Sat 27 Apr 2024 03:44:49 AM UTC. 2024-04-27T03:44:53.328 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:44:53.329 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:44:53.329 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2024-04-27T03:44:53.329 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:44:53.329 INFO:teuthology.orchestra.run.smithi073.stdout:Reinstalling: 2024-04-27T03:44:53.329 INFO:teuthology.orchestra.run.smithi073.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T03:44:53.329 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:44:53.329 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:44:53.329 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:44:53.329 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:44:53.329 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 255 k 2024-04-27T03:44:53.329 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 0 2024-04-27T03:44:53.330 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-04-27T03:44:53.698 INFO:teuthology.orchestra.run.smithi073.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.0 MB/s | 255 kB 00:00 2024-04-27T03:44:53.699 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:44:53.699 INFO:teuthology.orchestra.run.smithi073.stdout:Total 689 kB/s | 255 kB 00:00 2024-04-27T03:44:53.715 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-04-27T03:44:53.915 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-04-27T03:44:53.915 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-04-27T03:44:53.920 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-04-27T03:44:53.920 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-04-27T03:44:54.041 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-04-27T03:44:54.099 INFO:teuthology.orchestra.run.smithi073.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T03:44:54.100 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T03:44:54.726 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T03:44:54.726 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T03:44:55.474 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T03:44:55.474 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:44:55.475 INFO:teuthology.orchestra.run.smithi073.stdout:Reinstalled: 2024-04-27T03:44:55.475 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-27T03:44:55.475 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:44:55.475 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:44:55.644 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q kernel | sort -rV | head -n 1 2024-04-27T03:44:55.691 INFO:teuthology.orchestra.run.smithi073.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-27T03:44:55.691 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-27T03:44:55.691 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-27T03:44:55.691 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-27T03:44:55.691 DEBUG:teuthology.orchestra.run.smithi073:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-27T03:44:55.727 INFO:teuthology.orchestra.run.smithi073.stdout:ttyS1 2024-04-27T03:44:55.756 DEBUG:teuthology.parallel:result is None 2024-04-27T03:47:09.697 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 129 kB/s | 21 MB 02:46 2024-04-27T03:47:14.930 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-04-27T03:47:15.117 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:01 ago on Sat 27 Apr 2024 03:47:14 AM UTC. 2024-04-27T03:47:16.911 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-27T03:47:16.911 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-27T03:47:16.997 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:47:16.998 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-04-27T03:47:16.998 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:47:17.117 DEBUG:teuthology.orchestra.run.smithi046:> echo no | sudo yum reinstall kernel || true 2024-04-27T03:47:17.675 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:03 ago on Sat 27 Apr 2024 03:47:14 AM UTC. 2024-04-27T03:47:17.825 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:47:17.843 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:47:17.843 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-04-27T03:47:17.843 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:47:17.843 INFO:teuthology.orchestra.run.smithi046.stdout:Reinstalling: 2024-04-27T03:47:17.843 INFO:teuthology.orchestra.run.smithi046.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T03:47:17.843 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:47:17.843 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:47:17.843 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:47:17.843 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:47:17.843 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 255 k 2024-04-27T03:47:17.843 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 0 2024-04-27T03:47:17.844 INFO:teuthology.orchestra.run.smithi046.stderr:Operation aborted. 2024-04-27T03:47:17.915 INFO:teuthology.orchestra.run.smithi046.stdout:Is this ok [y/N]: 2024-04-27T03:47:17.915 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum reinstall -y kernel || true 2024-04-27T03:47:18.431 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:04 ago on Sat 27 Apr 2024 03:47:14 AM UTC. 2024-04-27T03:47:18.552 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:47:18.552 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:47:18.552 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-04-27T03:47:18.552 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:47:18.552 INFO:teuthology.orchestra.run.smithi046.stdout:Reinstalling: 2024-04-27T03:47:18.552 INFO:teuthology.orchestra.run.smithi046.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T03:47:18.552 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:47:18.552 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:47:18.552 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:47:18.553 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:47:18.553 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 255 k 2024-04-27T03:47:18.553 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 0 2024-04-27T03:47:18.553 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-27T03:47:18.915 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.1 MB/s | 255 kB 00:00 2024-04-27T03:47:18.916 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:47:18.916 INFO:teuthology.orchestra.run.smithi046.stdout:Total 702 kB/s | 255 kB 00:00 2024-04-27T03:47:18.933 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-27T03:47:19.124 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-27T03:47:19.125 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-27T03:47:19.130 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-27T03:47:19.130 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-27T03:47:19.269 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-27T03:47:19.341 INFO:teuthology.orchestra.run.smithi046.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T03:47:19.342 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T03:47:19.789 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T03:47:19.789 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T03:47:20.521 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T03:47:20.521 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:47:20.521 INFO:teuthology.orchestra.run.smithi046.stdout:Reinstalled: 2024-04-27T03:47:20.522 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-27T03:47:20.522 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:47:20.522 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:47:20.721 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-04-27T03:47:20.769 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-27T03:47:20.769 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-27T03:47:20.769 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-27T03:47:20.769 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-27T03:47:20.769 DEBUG:teuthology.orchestra.run.smithi046:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-27T03:47:20.800 INFO:teuthology.orchestra.run.smithi046.stdout:ttyS1 2024-04-27T03:47:20.818 DEBUG:teuthology.parallel:result is None 2024-04-27T03:47:20.818 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-27T03:47:20.825 INFO:teuthology.task.internal:Creating test directory... 2024-04-27T03:47:20.825 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-27T03:47:20.860 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-27T03:47:20.878 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-27T03:47:20.885 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-27T03:47:20.892 INFO:teuthology.task.internal:Creating archive directory... 2024-04-27T03:47:20.892 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-27T03:47:20.916 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-27T03:47:20.945 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-27T03:47:20.951 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-27T03:47:20.951 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-27T03:47:20.978 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-27T03:47:21.013 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T03:47:21.014 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T03:47:21.023 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T03:47:21.023 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T03:47:21.025 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-27T03:47:21.033 INFO:teuthology.task.internal:Configuring sudo... 2024-04-27T03:47:21.033 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-27T03:47:21.068 DEBUG:teuthology.orchestra.run.smithi073:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-27T03:47:21.093 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-27T03:47:21.102 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-27T03:47:21.102 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-27T03:47:21.132 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-27T03:47:21.149 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T03:47:21.227 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T03:47:21.277 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T03:47:21.358 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T03:47:21.398 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T03:47:21.398 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-27T03:47:21.462 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T03:47:21.502 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T03:47:21.563 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T03:47:21.644 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T03:47:21.689 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-04-27T03:47:21.689 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-27T03:47:21.755 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2024-04-27T03:47:21.757 DEBUG:teuthology.orchestra.run.smithi073:> sudo service rsyslog restart 2024-04-27T03:47:21.790 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T03:47:21.833 INFO:teuthology.orchestra.run.smithi073.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T03:47:22.215 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-27T03:47:22.223 INFO:teuthology.task.internal:Starting timer... 2024-04-27T03:47:22.223 INFO:teuthology.run_tasks:Running task pcp... 2024-04-27T03:47:22.233 INFO:teuthology.run_tasks:Running task selinux... 2024-04-27T03:47:22.242 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-27T03:47:22.242 DEBUG:teuthology.orchestra.run.smithi046:> sudo service auditd rotate 2024-04-27T03:47:22.302 INFO:teuthology.orchestra.run.smithi046.stdout:Rotating logs: 2024-04-27T03:47:22.304 DEBUG:teuthology.orchestra.run.smithi073:> sudo service auditd rotate 2024-04-27T03:47:22.358 INFO:teuthology.orchestra.run.smithi073.stdout:Rotating logs: 2024-04-27T03:47:22.359 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-27T03:47:22.359 DEBUG:teuthology.orchestra.run.smithi046:> /usr/sbin/getenforce 2024-04-27T03:47:22.383 INFO:teuthology.orchestra.run.smithi046.stdout:Permissive 2024-04-27T03:47:22.383 DEBUG:teuthology.orchestra.run.smithi073:> /usr/sbin/getenforce 2024-04-27T03:47:22.404 INFO:teuthology.orchestra.run.smithi073.stdout:Permissive 2024-04-27T03:47:22.404 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi046.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi073.front.sepia.ceph.com': 'permissive'} 2024-04-27T03:47:22.404 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T03:47:22.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T03:47:22.449 DEBUG:teuthology.orchestra.run.smithi073:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T03:47:22.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T03:47:22.474 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-27T03:47:22.474 DEBUG:teuthology.orchestra.run.smithi046:> sudo /usr/sbin/setenforce permissive 2024-04-27T03:47:22.519 DEBUG:teuthology.orchestra.run.smithi073:> sudo /usr/sbin/setenforce permissive 2024-04-27T03:47:22.542 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-27T03:47:22.551 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-27T03:47:22.557 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-27T03:47:22.686 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-27T03:47:22.702 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-27T03:47:22.703 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com,smithi073.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-27T03:50:53.430 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com'), Remote(name='ubuntu@smithi073.front.sepia.ceph.com')] 2024-04-27T03:50:53.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-04-27T03:50:53.432 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T03:50:53.509 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-04-27T03:50:53.589 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-04-27T03:50:53.589 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T03:50:53.590 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T03:50:53.662 DEBUG:teuthology.orchestra.run.smithi073:> true 2024-04-27T03:50:53.737 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T03:50:53.737 INFO:teuthology.run_tasks:Running task clock... 2024-04-27T03:50:53.752 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-27T03:50:53.753 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-27T03:50:53.753 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T03:50:53.756 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-27T03:50:53.756 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T03:50:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-27T03:50:53.795 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-27T03:50:53.804 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-27T03:50:53.818 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-27T03:50:53.818 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: ntpd: command not found 2024-04-27T03:50:53.829 INFO:teuthology.orchestra.run.smithi046.stdout:506 Cannot talk to daemon 2024-04-27T03:50:53.839 INFO:teuthology.orchestra.run.smithi073.stderr:sudo: ntpd: command not found 2024-04-27T03:50:53.842 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-27T03:50:53.850 INFO:teuthology.orchestra.run.smithi073.stdout:506 Cannot talk to daemon 2024-04-27T03:50:53.856 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-27T03:50:53.863 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-27T03:50:53.876 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-27T03:50:53.912 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-04-27T03:50:53.916 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T03:50:53.916 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-04-27T03:50:53.916 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T03:50:53.916 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T03:50:53.916 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T03:50:53.916 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T03:50:53.938 INFO:teuthology.orchestra.run.smithi073.stderr:bash: line 1: ntpq: command not found 2024-04-27T03:50:53.942 INFO:teuthology.orchestra.run.smithi073.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T03:50:53.942 INFO:teuthology.orchestra.run.smithi073.stdout:=============================================================================== 2024-04-27T03:50:53.942 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T03:50:53.942 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T03:50:53.942 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T03:50:53.942 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T03:50:53.943 INFO:teuthology.run_tasks:Running task install... 2024-04-27T03:50:53.954 DEBUG:teuthology.task.install:project ceph 2024-04-27T03:50:53.955 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4e213a58e787437ff508e6c890ccd5fd32dc96b5'}} 2024-04-27T03:50:53.955 DEBUG:teuthology.task.install:config {'branch': 'reef', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '4e213a58e787437ff508e6c890ccd5fd32dc96b5'} 2024-04-27T03:50:53.955 INFO:teuthology.task.install:Using flavor: default 2024-04-27T03:50:53.961 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-27T03:50:53.961 INFO:teuthology.task.install:extra packages: [] 2024-04-27T03:50:53.962 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e213a58e787437ff508e6c890ccd5fd32dc96b5', 'tag': None, 'wait_for_package': False} 2024-04-27T03:50:53.962 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-27T03:50:53.962 INFO:teuthology.packaging:ref: None 2024-04-27T03:50:53.962 INFO:teuthology.packaging:tag: None 2024-04-27T03:50:53.962 INFO:teuthology.packaging:branch: reef 2024-04-27T03:50:53.963 INFO:teuthology.packaging:sha1: 4e213a58e787437ff508e6c890ccd5fd32dc96b5 2024-04-27T03:50:53.963 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-04-27T03:50:53.965 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e213a58e787437ff508e6c890ccd5fd32dc96b5', 'tag': None, 'wait_for_package': False} 2024-04-27T03:50:53.965 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-27T03:50:53.965 INFO:teuthology.packaging:ref: None 2024-04-27T03:50:53.965 INFO:teuthology.packaging:tag: None 2024-04-27T03:50:53.965 INFO:teuthology.packaging:branch: reef 2024-04-27T03:50:53.965 INFO:teuthology.packaging:sha1: 4e213a58e787437ff508e6c890ccd5fd32dc96b5 2024-04-27T03:50:53.965 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-04-27T03:50:54.249 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/reef/5403a02127f27ec6c3309ef057522ab70cdeb9d2/centos/9/flavors/default/ 2024-04-27T03:50:54.250 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1260.g5403a021 2024-04-27T03:50:54.294 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/reef/5403a02127f27ec6c3309ef057522ab70cdeb9d2/centos/9/flavors/default/ 2024-04-27T03:50:54.294 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1260.g5403a021 2024-04-27T03:50:54.360 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/reef/5403a02127f27ec6c3309ef057522ab70cdeb9d2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/reef/5403a02127f27ec6c3309ef057522ab70cdeb9d2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/reef/5403a02127f27ec6c3309ef057522ab70cdeb9d2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-27T03:50:54.361 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-04-27T03:50:54.361 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-27T03:50:54.386 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-27T03:50:54.386 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-27T03:50:54.386 INFO:teuthology.packaging:ref: None 2024-04-27T03:50:54.386 INFO:teuthology.packaging:tag: None 2024-04-27T03:50:54.386 INFO:teuthology.packaging:branch: reef 2024-04-27T03:50:54.386 INFO:teuthology.packaging:sha1: 4e213a58e787437ff508e6c890ccd5fd32dc96b5 2024-04-27T03:50:54.386 DEBUG:teuthology.orchestra.run.smithi073:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-27T03:50:54.451 DEBUG:teuthology.orchestra.run.smithi073:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-27T03:50:54.463 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/reef/5403a02127f27ec6c3309ef057522ab70cdeb9d2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/reef/5403a02127f27ec6c3309ef057522ab70cdeb9d2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/reef/5403a02127f27ec6c3309ef057522ab70cdeb9d2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-27T03:50:54.464 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T03:50:54.464 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-27T03:50:54.491 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-27T03:50:54.491 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-27T03:50:54.491 INFO:teuthology.packaging:ref: None 2024-04-27T03:50:54.492 INFO:teuthology.packaging:tag: None 2024-04-27T03:50:54.492 INFO:teuthology.packaging:branch: reef 2024-04-27T03:50:54.492 INFO:teuthology.packaging:sha1: 4e213a58e787437ff508e6c890ccd5fd32dc96b5 2024-04-27T03:50:54.492 DEBUG:teuthology.orchestra.run.smithi046:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-27T03:50:54.525 DEBUG:teuthology.orchestra.run.smithi073:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-27T03:50:54.559 DEBUG:teuthology.orchestra.run.smithi046:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-27T03:50:54.588 INFO:teuthology.orchestra.run.smithi073.stdout:check_obsoletes = 1 2024-04-27T03:50:54.590 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum clean all 2024-04-27T03:50:54.637 DEBUG:teuthology.orchestra.run.smithi046:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-27T03:50:54.702 INFO:teuthology.orchestra.run.smithi046.stdout:check_obsoletes = 1 2024-04-27T03:50:54.704 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum clean all 2024-04-27T03:50:54.815 INFO:teuthology.orchestra.run.smithi073.stdout:48 files removed 2024-04-27T03:50:54.841 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-radosgw 2024-04-27T03:50:54.943 INFO:teuthology.orchestra.run.smithi046.stdout:48 files removed 2024-04-27T03:50:54.970 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-radosgw 2024-04-27T03:50:55.416 INFO:teuthology.orchestra.run.smithi073.stdout:ceph packages for x86_64 240 kB/s | 77 kB 00:00 2024-04-27T03:50:55.559 INFO:teuthology.orchestra.run.smithi046.stdout:ceph packages for x86_64 243 kB/s | 77 kB 00:00 2024-04-27T03:50:55.858 INFO:teuthology.orchestra.run.smithi073.stdout:ceph noarch packages 28 kB/s | 11 kB 00:00 2024-04-27T03:50:55.960 INFO:teuthology.orchestra.run.smithi046.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-04-27T03:50:56.167 INFO:teuthology.orchestra.run.smithi073.stdout:ceph source packages 6.9 kB/s | 1.9 kB 00:00 2024-04-27T03:50:56.294 INFO:teuthology.orchestra.run.smithi046.stdout:ceph source packages 6.3 kB/s | 1.9 kB 00:00 2024-04-27T03:50:56.925 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-27T03:50:57.986 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.0 MB 00:01 2024-04-27T03:50:58.743 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-04-27T03:50:59.762 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-04-27T03:51:04.079 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-27T03:51:04.311 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.2 MB 00:01 2024-04-27T03:51:05.906 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-04-27T03:51:06.695 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-04-27T03:51:08.404 INFO:teuthology.orchestra.run.smithi073.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-27T03:51:11.981 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 3.5 MB/s | 21 MB 00:05 2024-04-27T03:51:13.354 INFO:teuthology.orchestra.run.smithi073.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-27T03:51:15.498 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:51:15.500 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:51:15.501 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-04-27T03:51:15.501 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:51:15.501 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-04-27T03:51:15.501 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-radosgw x86_64 2:18.2.2-1260.g5403a021.el9 ceph 7.7 M 2024-04-27T03:51:15.501 INFO:teuthology.orchestra.run.smithi073.stdout:Upgrading: 2024-04-27T03:51:15.501 INFO:teuthology.orchestra.run.smithi073.stdout: librados2 x86_64 2:18.2.2-1260.g5403a021.el9 ceph 3.3 M 2024-04-27T03:51:15.502 INFO:teuthology.orchestra.run.smithi073.stdout: librbd1 x86_64 2:18.2.2-1260.g5403a021.el9 ceph 3.0 M 2024-04-27T03:51:15.502 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-04-27T03:51:15.502 INFO:teuthology.orchestra.run.smithi073.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-27T03:51:15.502 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-base x86_64 2:18.2.2-1260.g5403a021.el9 ceph 5.1 M 2024-04-27T03:51:15.502 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-common x86_64 2:18.2.2-1260.g5403a021.el9 ceph 18 M 2024-04-27T03:51:15.503 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-selinux x86_64 2:18.2.2-1260.g5403a021.el9 ceph 25 k 2024-04-27T03:51:15.503 INFO:teuthology.orchestra.run.smithi073.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-27T03:51:15.503 INFO:teuthology.orchestra.run.smithi073.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-27T03:51:15.503 INFO:teuthology.orchestra.run.smithi073.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-27T03:51:15.503 INFO:teuthology.orchestra.run.smithi073.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-27T03:51:15.503 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs2 x86_64 2:18.2.2-1260.g5403a021.el9 ceph 708 k 2024-04-27T03:51:15.503 INFO:teuthology.orchestra.run.smithi073.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-27T03:51:15.503 INFO:teuthology.orchestra.run.smithi073.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-27T03:51:15.504 INFO:teuthology.orchestra.run.smithi073.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-27T03:51:15.504 INFO:teuthology.orchestra.run.smithi073.stdout: libradosstriper1 x86_64 2:18.2.2-1260.g5403a021.el9 ceph 474 k 2024-04-27T03:51:15.504 INFO:teuthology.orchestra.run.smithi073.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-27T03:51:15.504 INFO:teuthology.orchestra.run.smithi073.stdout: librgw2 x86_64 2:18.2.2-1260.g5403a021.el9 ceph 4.5 M 2024-04-27T03:51:15.504 INFO:teuthology.orchestra.run.smithi073.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-27T03:51:15.504 INFO:teuthology.orchestra.run.smithi073.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-27T03:51:15.504 INFO:teuthology.orchestra.run.smithi073.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-27T03:51:15.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-argparse x86_64 2:18.2.2-1260.g5403a021.el9 ceph 45 k 2024-04-27T03:51:15.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-common x86_64 2:18.2.2-1260.g5403a021.el9 ceph 129 k 2024-04-27T03:51:15.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cephfs x86_64 2:18.2.2-1260.g5403a021.el9 ceph 162 k 2024-04-27T03:51:15.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-27T03:51:15.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rados x86_64 2:18.2.2-1260.g5403a021.el9 ceph 321 k 2024-04-27T03:51:15.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rbd x86_64 2:18.2.2-1260.g5403a021.el9 ceph 298 k 2024-04-27T03:51:15.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rgw x86_64 2:18.2.2-1260.g5403a021.el9 ceph 99 k 2024-04-27T03:51:15.505 INFO:teuthology.orchestra.run.smithi073.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-27T03:51:15.505 INFO:teuthology.orchestra.run.smithi073.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-27T03:51:15.506 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:51:15.506 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:51:15.506 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:51:15.506 INFO:teuthology.orchestra.run.smithi073.stdout:Install 28 Packages 2024-04-27T03:51:15.506 INFO:teuthology.orchestra.run.smithi073.stdout:Upgrade 2 Packages 2024-04-27T03:51:15.506 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:51:15.507 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 53 M 2024-04-27T03:51:15.507 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-04-27T03:51:16.997 INFO:teuthology.orchestra.run.smithi073.stdout:(1/30): ceph-radosgw-18.2.2-1260.g5403a021.el9. 7.6 MB/s | 7.7 MB 00:01 2024-04-27T03:51:17.038 INFO:teuthology.orchestra.run.smithi073.stdout:(2/30): ceph-selinux-18.2.2-1260.g5403a021.el9. 606 kB/s | 25 kB 00:00 2024-04-27T03:51:17.139 INFO:teuthology.orchestra.run.smithi073.stdout:(3/30): libcephfs2-18.2.2-1260.g5403a021.el9.x8 6.9 MB/s | 708 kB 00:00 2024-04-27T03:51:17.181 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-27T03:51:17.198 INFO:teuthology.orchestra.run.smithi073.stdout:(4/30): libradosstriper1-18.2.2-1260.g5403a021. 8.0 MB/s | 474 kB 00:00 2024-04-27T03:51:17.364 INFO:teuthology.orchestra.run.smithi073.stdout:(5/30): ceph-base-18.2.2-1260.g5403a021.el9.x86 3.7 MB/s | 5.1 MB 00:01 2024-04-27T03:51:17.451 INFO:teuthology.orchestra.run.smithi073.stdout:(6/30): python3-ceph-argparse-18.2.2-1260.g5403 540 kB/s | 45 kB 00:00 2024-04-27T03:51:17.573 INFO:teuthology.orchestra.run.smithi073.stdout:(7/30): librgw2-18.2.2-1260.g5403a021.el9.x86_6 12 MB/s | 4.5 MB 00:00 2024-04-27T03:51:17.598 INFO:teuthology.orchestra.run.smithi073.stdout:(8/30): python3-ceph-common-18.2.2-1260.g5403a0 860 kB/s | 129 kB 00:00 2024-04-27T03:51:17.624 INFO:teuthology.orchestra.run.smithi073.stdout:(9/30): python3-cephfs-18.2.2-1260.g5403a021.el 3.1 MB/s | 162 kB 00:00 2024-04-27T03:51:17.674 INFO:teuthology.orchestra.run.smithi073.stdout:(10/30): python3-rbd-18.2.2-1260.g5403a021.el9. 5.8 MB/s | 298 kB 00:00 2024-04-27T03:51:17.700 INFO:teuthology.orchestra.run.smithi073.stdout:(11/30): python3-rados-18.2.2-1260.g5403a021.el 3.1 MB/s | 321 kB 00:00 2024-04-27T03:51:17.725 INFO:teuthology.orchestra.run.smithi073.stdout:(12/30): python3-rgw-18.2.2-1260.g5403a021.el9. 1.9 MB/s | 99 kB 00:00 2024-04-27T03:51:17.842 INFO:teuthology.orchestra.run.smithi073.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 753 kB/s | 107 kB 00:00 2024-04-27T03:51:17.893 INFO:teuthology.orchestra.run.smithi073.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 959 kB/s | 160 kB 00:00 2024-04-27T03:51:17.918 INFO:teuthology.orchestra.run.smithi073.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 601 kB/s | 45 kB 00:00 2024-04-27T03:51:18.035 INFO:teuthology.orchestra.run.smithi073.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-04-27T03:51:18.085 INFO:teuthology.orchestra.run.smithi073.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 847 kB/s | 42 kB 00:00 2024-04-27T03:51:18.118 INFO:teuthology.orchestra.run.smithi073.stdout:(18/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.9 MB/s | 662 kB 00:00 2024-04-27T03:51:18.310 INFO:teuthology.orchestra.run.smithi073.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 491 kB/s | 111 kB 00:00 2024-04-27T03:51:18.352 INFO:teuthology.orchestra.run.smithi073.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2024-04-27T03:51:18.544 INFO:teuthology.orchestra.run.smithi073.stdout:(21/30): ceph-common-18.2.2-1260.g5403a021.el9. 7.1 MB/s | 18 MB 00:02 2024-04-27T03:51:18.561 INFO:teuthology.orchestra.run.smithi073.stdout:(22/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 124 kB/s | 26 kB 00:00 2024-04-27T03:51:18.612 INFO:teuthology.orchestra.run.smithi073.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-27T03:51:18.704 INFO:teuthology.orchestra.run.smithi073.stdout:(24/30): liboath-2.6.7-2.el9.x86_64.rpm 306 kB/s | 49 kB 00:00 2024-04-27T03:51:18.795 INFO:teuthology.orchestra.run.smithi073.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-27T03:51:18.821 INFO:teuthology.orchestra.run.smithi073.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.9 MB/s | 839 kB 00:00 2024-04-27T03:51:19.113 INFO:teuthology.orchestra.run.smithi073.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 5.0 MB/s | 1.6 MB 00:00 2024-04-27T03:51:19.321 INFO:teuthology.orchestra.run.smithi073.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:01 2024-04-27T03:51:19.411 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:51:19.413 INFO:teuthology.orchestra.run.smithi073.stdout:(29/30): librados2-18.2.2-1260.g5403a021.el9.x8 5.6 MB/s | 3.3 MB 00:00 2024-04-27T03:51:19.413 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:51:19.413 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-04-27T03:51:19.413 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:51:19.413 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-27T03:51:19.413 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw x86_64 2:18.2.2-1260.g5403a021.el9 ceph 7.7 M 2024-04-27T03:51:19.413 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrading: 2024-04-27T03:51:19.414 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 x86_64 2:18.2.2-1260.g5403a021.el9 ceph 3.3 M 2024-04-27T03:51:19.414 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1 x86_64 2:18.2.2-1260.g5403a021.el9 ceph 3.0 M 2024-04-27T03:51:19.414 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-04-27T03:51:19.414 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-27T03:51:19.414 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base x86_64 2:18.2.2-1260.g5403a021.el9 ceph 5.1 M 2024-04-27T03:51:19.414 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common x86_64 2:18.2.2-1260.g5403a021.el9 ceph 18 M 2024-04-27T03:51:19.414 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux x86_64 2:18.2.2-1260.g5403a021.el9 ceph 25 k 2024-04-27T03:51:19.414 INFO:teuthology.orchestra.run.smithi046.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-27T03:51:19.414 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-27T03:51:19.414 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-27T03:51:19.414 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-27T03:51:19.414 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2 x86_64 2:18.2.2-1260.g5403a021.el9 ceph 708 k 2024-04-27T03:51:19.414 INFO:teuthology.orchestra.run.smithi046.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-27T03:51:19.415 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-27T03:51:19.415 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-27T03:51:19.415 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 x86_64 2:18.2.2-1260.g5403a021.el9 ceph 474 k 2024-04-27T03:51:19.415 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-27T03:51:19.415 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2 x86_64 2:18.2.2-1260.g5403a021.el9 ceph 4.5 M 2024-04-27T03:51:19.415 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-27T03:51:19.415 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-27T03:51:19.415 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-27T03:51:19.415 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse x86_64 2:18.2.2-1260.g5403a021.el9 ceph 45 k 2024-04-27T03:51:19.415 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common x86_64 2:18.2.2-1260.g5403a021.el9 ceph 129 k 2024-04-27T03:51:19.415 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs x86_64 2:18.2.2-1260.g5403a021.el9 ceph 162 k 2024-04-27T03:51:19.415 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-27T03:51:19.415 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados x86_64 2:18.2.2-1260.g5403a021.el9 ceph 321 k 2024-04-27T03:51:19.415 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd x86_64 2:18.2.2-1260.g5403a021.el9 ceph 298 k 2024-04-27T03:51:19.416 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw x86_64 2:18.2.2-1260.g5403a021.el9 ceph 99 k 2024-04-27T03:51:19.416 INFO:teuthology.orchestra.run.smithi046.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-27T03:51:19.416 INFO:teuthology.orchestra.run.smithi046.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-27T03:51:19.416 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:51:19.416 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:51:19.416 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:51:19.416 INFO:teuthology.orchestra.run.smithi046.stdout:Install 28 Packages 2024-04-27T03:51:19.416 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrade 2 Packages 2024-04-27T03:51:19.416 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:51:19.417 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 53 M 2024-04-27T03:51:19.417 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-27T03:51:19.480 INFO:teuthology.orchestra.run.smithi073.stdout:(30/30): librbd1-18.2.2-1260.g5403a021.el9.x86_ 8.2 MB/s | 3.0 MB 00:00 2024-04-27T03:51:19.484 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:51:19.484 INFO:teuthology.orchestra.run.smithi073.stdout:Total 13 MB/s | 53 MB 00:03 2024-04-27T03:51:19.551 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-04-27T03:51:19.585 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-04-27T03:51:19.585 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-04-27T03:51:20.004 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-04-27T03:51:20.004 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-04-27T03:51:21.099 INFO:teuthology.orchestra.run.smithi046.stdout:(1/30): ceph-common-18.2.2-1260.g5403a021.el9.x 16 MB/s | 18 MB 00:01 2024-04-27T03:51:21.149 INFO:teuthology.orchestra.run.smithi046.stdout:(2/30): ceph-selinux-18.2.2-1260.g5403a021.el9. 505 kB/s | 25 kB 00:00 2024-04-27T03:51:21.232 INFO:teuthology.orchestra.run.smithi046.stdout:(3/30): libcephfs2-18.2.2-1260.g5403a021.el9.x8 8.3 MB/s | 708 kB 00:00 2024-04-27T03:51:21.291 INFO:teuthology.orchestra.run.smithi046.stdout:(4/30): libradosstriper1-18.2.2-1260.g5403a021. 8.0 MB/s | 474 kB 00:00 2024-04-27T03:51:21.396 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-04-27T03:51:21.481 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-27T03:51:21.509 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-27T03:51:21.559 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-27T03:51:21.591 INFO:teuthology.orchestra.run.smithi046.stdout:(5/30): librgw2-18.2.2-1260.g5403a021.el9.x86_6 15 MB/s | 4.5 MB 00:00 2024-04-27T03:51:21.633 INFO:teuthology.orchestra.run.smithi046.stdout:(6/30): python3-ceph-argparse-18.2.2-1260.g5403 1.1 MB/s | 45 kB 00:00 2024-04-27T03:51:21.700 INFO:teuthology.orchestra.run.smithi046.stdout:(7/30): ceph-base-18.2.2-1260.g5403a021.el9.x86 2.9 MB/s | 5.1 MB 00:01 2024-04-27T03:51:21.725 INFO:teuthology.orchestra.run.smithi046.stdout:(8/30): python3-ceph-common-18.2.2-1260.g5403a0 1.4 MB/s | 129 kB 00:00 2024-04-27T03:51:21.759 INFO:teuthology.orchestra.run.smithi046.stdout:(9/30): python3-cephfs-18.2.2-1260.g5403a021.el 2.7 MB/s | 162 kB 00:00 2024-04-27T03:51:21.784 INFO:teuthology.orchestra.run.smithi046.stdout:(10/30): python3-rados-18.2.2-1260.g5403a021.el 5.3 MB/s | 321 kB 00:00 2024-04-27T03:51:21.826 INFO:teuthology.orchestra.run.smithi046.stdout:(11/30): python3-rbd-18.2.2-1260.g5403a021.el9. 4.4 MB/s | 298 kB 00:00 2024-04-27T03:51:21.852 INFO:teuthology.orchestra.run.smithi046.stdout:(12/30): python3-rgw-18.2.2-1260.g5403a021.el9. 1.5 MB/s | 99 kB 00:00 2024-04-27T03:51:21.933 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-27T03:51:21.994 INFO:teuthology.orchestra.run.smithi046.stdout:(13/30): ceph-radosgw-18.2.2-1260.g5403a021.el9 3.7 MB/s | 7.7 MB 00:02 2024-04-27T03:51:22.009 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : librados2-2:18.2.2-1260.g5403a021.el9.x86_64 5/32 2024-04-27T03:51:22.101 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados2-2:18.2.2-1260.g5403a021.el9.x86_64 5/32 2024-04-27T03:51:22.184 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rados-2:18.2.2-1260.g5403a021.el9.x86_64 6/32 2024-04-27T03:51:22.210 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephfs2-2:18.2.2-1260.g5403a021.el9.x86_64 7/32 2024-04-27T03:51:22.272 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephfs2-2:18.2.2-1260.g5403a021.el9.x86_64 7/32 2024-04-27T03:51:22.301 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-27T03:51:22.345 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ceph-argparse-2:18.2.2-1260.g5403a021.el9. 9/32 2024-04-27T03:51:22.402 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cephfs-2:18.2.2-1260.g5403a021.el9.x86_64 10/32 2024-04-27T03:51:22.429 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libradosstriper1-2:18.2.2-1260.g5403a021.el9.x86_6 11/32 2024-04-27T03:51:22.494 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1260.g5403a021.el9.x86_6 11/32 2024-04-27T03:51:22.600 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-27T03:51:22.628 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-04-27T03:51:22.652 INFO:teuthology.orchestra.run.smithi046.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 129 kB/s | 107 kB 00:00 2024-04-27T03:51:22.659 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-27T03:51:22.677 INFO:teuthology.orchestra.run.smithi046.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 66 kB/s | 45 kB 00:00 2024-04-27T03:51:22.684 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-27T03:51:22.712 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-04-27T03:51:22.748 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-27T03:51:22.781 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-27T03:51:22.986 INFO:teuthology.orchestra.run.smithi046.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 141 kB/s | 160 kB 00:01 2024-04-27T03:51:23.080 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-27T03:51:23.119 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : librbd1-2:18.2.2-1260.g5403a021.el9.x86_64 20/32 2024-04-27T03:51:23.200 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librbd1-2:18.2.2-1260.g5403a021.el9.x86_64 20/32 2024-04-27T03:51:23.231 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rbd-2:18.2.2-1260.g5403a021.el9.x86_64 21/32 2024-04-27T03:51:23.267 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-27T03:51:23.645 INFO:teuthology.orchestra.run.smithi046.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 64 kB/s | 42 kB 00:00 2024-04-27T03:51:23.671 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-04-27T03:51:23.696 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librgw2-2:18.2.2-1260.g5403a021.el9.x86_64 24/32 2024-04-27T03:51:23.767 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librgw2-2:18.2.2-1260.g5403a021.el9.x86_64 24/32 2024-04-27T03:51:23.770 INFO:teuthology.orchestra.run.smithi046.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 268 kB/s | 292 kB 00:01 2024-04-27T03:51:23.836 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rgw-2:18.2.2-1260.g5403a021.el9.x86_64 25/32 2024-04-27T03:51:23.865 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ceph-common-2:18.2.2-1260.g5403a021.el9.x8 26/32 2024-04-27T03:51:23.945 INFO:teuthology.orchestra.run.smithi046.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 369 kB/s | 111 kB 00:00 2024-04-27T03:51:24.087 INFO:teuthology.orchestra.run.smithi046.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 976 kB/s | 309 kB 00:00 2024-04-27T03:51:24.146 INFO:teuthology.orchestra.run.smithi046.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 447 kB/s | 26 kB 00:00 2024-04-27T03:51:24.204 INFO:teuthology.orchestra.run.smithi046.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 836 kB/s | 49 kB 00:00 2024-04-27T03:51:24.263 INFO:teuthology.orchestra.run.smithi046.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-27T03:51:24.355 INFO:teuthology.orchestra.run.smithi046.stdout:(24/30): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-27T03:51:24.380 INFO:teuthology.orchestra.run.smithi046.stdout:(25/30): librdkafka-1.6.1-102.el9.x86_64.rpm 383 kB/s | 662 kB 00:01 2024-04-27T03:51:24.413 INFO:teuthology.orchestra.run.smithi046.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.5 MB/s | 839 kB 00:00 2024-04-27T03:51:24.464 INFO:teuthology.orchestra.run.smithi046.stdout:(27/30): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-04-27T03:51:24.739 INFO:teuthology.orchestra.run.smithi046.stdout:(28/30): librbd1-18.2.2-1260.g5403a021.el9.x86_ 11 MB/s | 3.0 MB 00:00 2024-04-27T03:51:24.797 INFO:teuthology.orchestra.run.smithi046.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 3.8 MB/s | 1.6 MB 00:00 2024-04-27T03:51:25.073 INFO:teuthology.orchestra.run.smithi046.stdout:(30/30): librados2-18.2.2-1260.g5403a021.el9.x8 5.2 MB/s | 3.3 MB 00:00 2024-04-27T03:51:25.079 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:51:25.079 INFO:teuthology.orchestra.run.smithi046.stdout:Total 9.3 MB/s | 53 MB 00:05 2024-04-27T03:51:25.151 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-27T03:51:25.192 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-27T03:51:25.192 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-27T03:51:25.639 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-27T03:51:25.639 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-27T03:51:25.961 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-common-2:18.2.2-1260.g5403a021.el9.x86_64 27/32 2024-04-27T03:51:26.004 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-common-2:18.2.2-1260.g5403a021.el9.x86_64 27/32 2024-04-27T03:51:26.520 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-common-2:18.2.2-1260.g5403a021.el9.x86_64 27/32 2024-04-27T03:51:26.557 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-base-2:18.2.2-1260.g5403a021.el9.x86_64 28/32 2024-04-27T03:51:26.648 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-base-2:18.2.2-1260.g5403a021.el9.x86_64 28/32 2024-04-27T03:51:26.649 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-27T03:51:26.649 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-27T03:51:26.649 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:51:26.685 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-selinux-2:18.2.2-1260.g5403a021.el9.x86_64 29/32 2024-04-27T03:51:27.034 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-27T03:51:27.124 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-27T03:51:27.161 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-27T03:51:27.212 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-27T03:51:27.608 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-27T03:51:27.677 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librados2-2:18.2.2-1260.g5403a021.el9.x86_64 5/32 2024-04-27T03:51:27.770 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:18.2.2-1260.g5403a021.el9.x86_64 5/32 2024-04-27T03:51:27.856 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rados-2:18.2.2-1260.g5403a021.el9.x86_64 6/32 2024-04-27T03:51:27.886 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs2-2:18.2.2-1260.g5403a021.el9.x86_64 7/32 2024-04-27T03:51:27.957 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs2-2:18.2.2-1260.g5403a021.el9.x86_64 7/32 2024-04-27T03:51:27.986 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-27T03:51:28.030 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-argparse-2:18.2.2-1260.g5403a021.el9. 9/32 2024-04-27T03:51:28.097 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cephfs-2:18.2.2-1260.g5403a021.el9.x86_64 10/32 2024-04-27T03:51:28.122 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libradosstriper1-2:18.2.2-1260.g5403a021.el9.x86_6 11/32 2024-04-27T03:51:28.187 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1260.g5403a021.el9.x86_6 11/32 2024-04-27T03:51:28.307 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-27T03:51:28.337 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-04-27T03:51:28.377 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-27T03:51:28.411 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-27T03:51:28.438 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-04-27T03:51:28.474 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-27T03:51:28.507 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-27T03:51:28.822 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-27T03:51:28.861 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librbd1-2:18.2.2-1260.g5403a021.el9.x86_64 20/32 2024-04-27T03:51:28.952 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:18.2.2-1260.g5403a021.el9.x86_64 20/32 2024-04-27T03:51:28.982 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rbd-2:18.2.2-1260.g5403a021.el9.x86_64 21/32 2024-04-27T03:51:29.019 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-27T03:51:29.441 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-04-27T03:51:29.471 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librgw2-2:18.2.2-1260.g5403a021.el9.x86_64 24/32 2024-04-27T03:51:29.543 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librgw2-2:18.2.2-1260.g5403a021.el9.x86_64 24/32 2024-04-27T03:51:29.622 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rgw-2:18.2.2-1260.g5403a021.el9.x86_64 25/32 2024-04-27T03:51:29.657 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-common-2:18.2.2-1260.g5403a021.el9.x8 26/32 2024-04-27T03:51:31.786 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:18.2.2-1260.g5403a021.el9.x86_64 27/32 2024-04-27T03:51:31.822 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-common-2:18.2.2-1260.g5403a021.el9.x86_64 27/32 2024-04-27T03:51:32.369 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:18.2.2-1260.g5403a021.el9.x86_64 27/32 2024-04-27T03:51:32.408 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-base-2:18.2.2-1260.g5403a021.el9.x86_64 28/32 2024-04-27T03:51:32.505 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-base-2:18.2.2-1260.g5403a021.el9.x86_64 28/32 2024-04-27T03:51:32.505 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-27T03:51:32.505 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-27T03:51:32.505 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:51:32.643 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-selinux-2:18.2.2-1260.g5403a021.el9.x86_64 29/32 2024-04-27T03:51:35.956 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1260.g5403a021.el9.x86_64 29/32 2024-04-27T03:51:35.956 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /sys 2024-04-27T03:51:35.956 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /proc 2024-04-27T03:51:35.956 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /mnt 2024-04-27T03:51:35.956 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /var/tmp 2024-04-27T03:51:35.956 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /home 2024-04-27T03:51:35.956 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /root 2024-04-27T03:51:35.957 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /tmp 2024-04-27T03:51:35.957 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:51:36.709 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-radosgw-2:18.2.2-1260.g5403a021.el9.x86_64 30/32 2024-04-27T03:51:36.732 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1260.g5403a021.el9.x86_64 30/32 2024-04-27T03:51:36.733 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T03:51:36.733 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-27T03:51:36.733 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-27T03:51:36.733 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-27T03:51:36.733 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:51:36.735 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-27T03:51:36.811 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-27T03:51:36.812 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-27T03:51:37.999 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-27T03:51:37.999 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-base-2:18.2.2-1260.g5403a021.el9.x86_64 1/32 2024-04-27T03:51:38.000 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-common-2:18.2.2-1260.g5403a021.el9.x86_64 2/32 2024-04-27T03:51:38.000 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-radosgw-2:18.2.2-1260.g5403a021.el9.x86_64 3/32 2024-04-27T03:51:38.000 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-selinux-2:18.2.2-1260.g5403a021.el9.x86_64 4/32 2024-04-27T03:51:38.000 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephfs2-2:18.2.2-1260.g5403a021.el9.x86_64 5/32 2024-04-27T03:51:38.000 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libradosstriper1-2:18.2.2-1260.g5403a021.el9.x86_6 6/32 2024-04-27T03:51:38.000 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librgw2-2:18.2.2-1260.g5403a021.el9.x86_64 7/32 2024-04-27T03:51:38.000 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1260.g5403a021.el9. 8/32 2024-04-27T03:51:38.000 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ceph-common-2:18.2.2-1260.g5403a021.el9.x8 9/32 2024-04-27T03:51:38.000 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cephfs-2:18.2.2-1260.g5403a021.el9.x86_64 10/32 2024-04-27T03:51:38.000 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rados-2:18.2.2-1260.g5403a021.el9.x86_64 11/32 2024-04-27T03:51:38.000 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rbd-2:18.2.2-1260.g5403a021.el9.x86_64 12/32 2024-04-27T03:51:38.000 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rgw-2:18.2.2-1260.g5403a021.el9.x86_64 13/32 2024-04-27T03:51:38.000 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-27T03:51:38.001 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-27T03:51:38.001 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-27T03:51:38.001 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-27T03:51:38.001 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-27T03:51:38.001 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-27T03:51:38.001 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-27T03:51:38.001 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-27T03:51:38.001 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-04-27T03:51:38.001 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-04-27T03:51:38.001 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-27T03:51:38.001 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-27T03:51:38.001 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-04-27T03:51:38.001 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-27T03:51:38.001 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-27T03:51:38.002 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados2-2:18.2.2-1260.g5403a021.el9.x86_64 29/32 2024-04-27T03:51:38.002 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-27T03:51:38.002 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librbd1-2:18.2.2-1260.g5403a021.el9.x86_64 31/32 2024-04-27T03:51:38.738 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-27T03:51:38.739 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:51:38.739 INFO:teuthology.orchestra.run.smithi073.stdout:Upgraded: 2024-04-27T03:51:38.739 INFO:teuthology.orchestra.run.smithi073.stdout: librados2-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:38.739 INFO:teuthology.orchestra.run.smithi073.stdout: librbd1-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:38.739 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-04-27T03:51:38.739 INFO:teuthology.orchestra.run.smithi073.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-27T03:51:38.739 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-base-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:38.739 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-common-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:38.739 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-radosgw-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:38.739 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-selinux-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:38.739 INFO:teuthology.orchestra.run.smithi073.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-27T03:51:38.739 INFO:teuthology.orchestra.run.smithi073.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-27T03:51:38.740 INFO:teuthology.orchestra.run.smithi073.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-27T03:51:38.740 INFO:teuthology.orchestra.run.smithi073.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-27T03:51:38.740 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs2-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:38.740 INFO:teuthology.orchestra.run.smithi073.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-27T03:51:38.740 INFO:teuthology.orchestra.run.smithi073.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-27T03:51:38.740 INFO:teuthology.orchestra.run.smithi073.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-27T03:51:38.740 INFO:teuthology.orchestra.run.smithi073.stdout: libradosstriper1-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:38.740 INFO:teuthology.orchestra.run.smithi073.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-27T03:51:38.740 INFO:teuthology.orchestra.run.smithi073.stdout: librgw2-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:38.740 INFO:teuthology.orchestra.run.smithi073.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-27T03:51:38.740 INFO:teuthology.orchestra.run.smithi073.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-27T03:51:38.740 INFO:teuthology.orchestra.run.smithi073.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-27T03:51:38.740 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-argparse-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:38.740 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-common-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:38.741 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cephfs-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:38.741 INFO:teuthology.orchestra.run.smithi073.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-27T03:51:38.741 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rados-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:38.741 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rbd-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:38.741 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rgw-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:38.741 INFO:teuthology.orchestra.run.smithi073.stdout: re2-1:20211101-3.el9.x86_64 2024-04-27T03:51:38.741 INFO:teuthology.orchestra.run.smithi073.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-27T03:51:38.741 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:51:38.741 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:51:39.016 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-test 2024-04-27T03:51:39.546 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:26 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:51:39.668 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:51:39.668 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:51:39.668 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-04-27T03:51:39.669 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:51:39.669 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-04-27T03:51:39.669 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-test x86_64 2:18.2.2-1260.g5403a021.el9 ceph 40 M 2024-04-27T03:51:39.669 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-04-27T03:51:39.669 INFO:teuthology.orchestra.run.smithi073.stdout: libcephsqlite x86_64 2:18.2.2-1260.g5403a021.el9 ceph 166 k 2024-04-27T03:51:39.669 INFO:teuthology.orchestra.run.smithi073.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-27T03:51:39.669 INFO:teuthology.orchestra.run.smithi073.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-27T03:51:39.669 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:51:39.669 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:51:39.669 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:51:39.669 INFO:teuthology.orchestra.run.smithi073.stdout:Install 4 Packages 2024-04-27T03:51:39.669 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:51:39.670 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 40 M 2024-04-27T03:51:39.670 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 163 M 2024-04-27T03:51:39.670 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-04-27T03:51:40.179 INFO:teuthology.orchestra.run.smithi073.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 305 kB 00:00 2024-04-27T03:51:40.230 INFO:teuthology.orchestra.run.smithi073.stdout:(2/4): libcephsqlite-18.2.2-1260.g5403a021.el9. 576 kB/s | 166 kB 00:00 2024-04-27T03:51:40.263 INFO:teuthology.orchestra.run.smithi073.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 761 kB/s | 64 kB 00:00 2024-04-27T03:51:42.015 INFO:teuthology.orchestra.run.smithi073.stdout:(4/4): ceph-test-18.2.2-1260.g5403a021.el9.x86_ 19 MB/s | 40 MB 00:02 2024-04-27T03:51:42.017 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:51:42.018 INFO:teuthology.orchestra.run.smithi073.stdout:Total 17 MB/s | 40 MB 00:02 2024-04-27T03:51:42.045 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-04-27T03:51:42.065 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-04-27T03:51:42.065 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-04-27T03:51:42.378 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-04-27T03:51:42.378 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-04-27T03:51:42.555 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1260.g5403a021.el9.x86_64 29/32 2024-04-27T03:51:42.555 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /sys 2024-04-27T03:51:42.556 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /proc 2024-04-27T03:51:42.556 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /mnt 2024-04-27T03:51:42.556 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /var/tmp 2024-04-27T03:51:42.556 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /home 2024-04-27T03:51:42.556 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /root 2024-04-27T03:51:42.556 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /tmp 2024-04-27T03:51:42.556 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:51:42.838 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-04-27T03:51:42.916 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-27T03:51:42.955 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-27T03:51:42.983 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephsqlite-2:18.2.2-1260.g5403a021.el9.x86_64 3/4 2024-04-27T03:51:43.307 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-radosgw-2:18.2.2-1260.g5403a021.el9.x86_64 30/32 2024-04-27T03:51:43.332 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1260.g5403a021.el9.x86_64 30/32 2024-04-27T03:51:43.332 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T03:51:43.332 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-27T03:51:43.332 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-27T03:51:43.333 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-27T03:51:43.333 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:51:43.334 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-27T03:51:43.401 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-27T03:51:43.401 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-27T03:51:44.567 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-27T03:51:44.567 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-base-2:18.2.2-1260.g5403a021.el9.x86_64 1/32 2024-04-27T03:51:44.567 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-common-2:18.2.2-1260.g5403a021.el9.x86_64 2/32 2024-04-27T03:51:44.567 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-radosgw-2:18.2.2-1260.g5403a021.el9.x86_64 3/32 2024-04-27T03:51:44.567 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-selinux-2:18.2.2-1260.g5403a021.el9.x86_64 4/32 2024-04-27T03:51:44.567 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs2-2:18.2.2-1260.g5403a021.el9.x86_64 5/32 2024-04-27T03:51:44.567 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libradosstriper1-2:18.2.2-1260.g5403a021.el9.x86_6 6/32 2024-04-27T03:51:44.567 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librgw2-2:18.2.2-1260.g5403a021.el9.x86_64 7/32 2024-04-27T03:51:44.567 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1260.g5403a021.el9. 8/32 2024-04-27T03:51:44.567 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-common-2:18.2.2-1260.g5403a021.el9.x8 9/32 2024-04-27T03:51:44.568 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cephfs-2:18.2.2-1260.g5403a021.el9.x86_64 10/32 2024-04-27T03:51:44.568 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rados-2:18.2.2-1260.g5403a021.el9.x86_64 11/32 2024-04-27T03:51:44.568 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rbd-2:18.2.2-1260.g5403a021.el9.x86_64 12/32 2024-04-27T03:51:44.568 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rgw-2:18.2.2-1260.g5403a021.el9.x86_64 13/32 2024-04-27T03:51:44.568 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-27T03:51:44.568 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-27T03:51:44.568 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-27T03:51:44.568 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-27T03:51:44.568 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-27T03:51:44.568 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-27T03:51:44.568 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-27T03:51:44.568 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-27T03:51:44.568 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-04-27T03:51:44.569 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-04-27T03:51:44.569 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-27T03:51:44.569 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-27T03:51:44.569 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-04-27T03:51:44.569 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-27T03:51:44.569 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-27T03:51:44.569 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:18.2.2-1260.g5403a021.el9.x86_64 29/32 2024-04-27T03:51:44.569 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-27T03:51:44.569 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:18.2.2-1260.g5403a021.el9.x86_64 31/32 2024-04-27T03:51:45.354 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-27T03:51:45.354 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:51:45.354 INFO:teuthology.orchestra.run.smithi046.stdout:Upgraded: 2024-04-27T03:51:45.354 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:45.355 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:45.355 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-27T03:51:45.355 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-27T03:51:45.355 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:45.355 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:45.355 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:45.355 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:45.355 INFO:teuthology.orchestra.run.smithi046.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-27T03:51:45.355 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-27T03:51:45.355 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-27T03:51:45.355 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-27T03:51:45.355 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:45.355 INFO:teuthology.orchestra.run.smithi046.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-27T03:51:45.356 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-27T03:51:45.356 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-27T03:51:45.356 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:45.356 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-27T03:51:45.356 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:45.356 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-27T03:51:45.356 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-27T03:51:45.356 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-27T03:51:45.356 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:45.356 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:45.356 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:45.356 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-27T03:51:45.356 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:45.357 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:45.357 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:45.357 INFO:teuthology.orchestra.run.smithi046.stdout: re2-1:20211101-3.el9.x86_64 2024-04-27T03:51:45.357 INFO:teuthology.orchestra.run.smithi046.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-27T03:51:45.357 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:51:45.357 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:51:45.623 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-test 2024-04-27T03:51:46.241 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:29 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:51:46.370 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:51:46.370 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:51:46.370 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-04-27T03:51:46.371 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:51:46.371 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-27T03:51:46.371 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test x86_64 2:18.2.2-1260.g5403a021.el9 ceph 40 M 2024-04-27T03:51:46.371 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-04-27T03:51:46.371 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite x86_64 2:18.2.2-1260.g5403a021.el9 ceph 166 k 2024-04-27T03:51:46.371 INFO:teuthology.orchestra.run.smithi046.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-27T03:51:46.371 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-27T03:51:46.371 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:51:46.371 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:51:46.371 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:51:46.371 INFO:teuthology.orchestra.run.smithi046.stdout:Install 4 Packages 2024-04-27T03:51:46.371 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:51:46.371 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 40 M 2024-04-27T03:51:46.372 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 163 M 2024-04-27T03:51:46.372 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-27T03:51:46.640 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1260.g5403a021.el9.x86_64 3/4 2024-04-27T03:51:46.713 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-test-2:18.2.2-1260.g5403a021.el9.x86_64 4/4 2024-04-27T03:51:46.869 INFO:teuthology.orchestra.run.smithi046.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 305 kB 00:00 2024-04-27T03:51:46.919 INFO:teuthology.orchestra.run.smithi046.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-27T03:51:47.063 INFO:teuthology.orchestra.run.smithi046.stdout:(3/4): libcephsqlite-18.2.2-1260.g5403a021.el9. 408 kB/s | 166 kB 00:00 2024-04-27T03:51:48.622 INFO:teuthology.orchestra.run.smithi046.stdout:(4/4): ceph-test-18.2.2-1260.g5403a021.el9.x86_ 20 MB/s | 40 MB 00:01 2024-04-27T03:51:48.624 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:51:48.624 INFO:teuthology.orchestra.run.smithi046.stdout:Total 18 MB/s | 40 MB 00:02 2024-04-27T03:51:48.651 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-27T03:51:48.673 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-27T03:51:48.673 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-27T03:51:48.770 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-test-2:18.2.2-1260.g5403a021.el9.x86_64 4/4 2024-04-27T03:51:48.770 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-test-2:18.2.2-1260.g5403a021.el9.x86_64 1/4 2024-04-27T03:51:48.770 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephsqlite-2:18.2.2-1260.g5403a021.el9.x86_64 2/4 2024-04-27T03:51:48.770 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-27T03:51:49.005 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-27T03:51:49.005 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-27T03:51:49.070 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-27T03:51:49.070 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:51:49.070 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-04-27T03:51:49.070 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-test-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:49.070 INFO:teuthology.orchestra.run.smithi073.stdout: libcephsqlite-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:49.071 INFO:teuthology.orchestra.run.smithi073.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-27T03:51:49.071 INFO:teuthology.orchestra.run.smithi073.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-27T03:51:49.071 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:51:49.071 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:51:49.314 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph 2024-04-27T03:51:49.486 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-27T03:51:49.565 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-27T03:51:49.605 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-27T03:51:49.632 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephsqlite-2:18.2.2-1260.g5403a021.el9.x86_64 3/4 2024-04-27T03:51:49.823 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:36 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:51:49.948 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:51:49.950 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:51:49.950 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-04-27T03:51:49.951 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:51:49.951 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-04-27T03:51:49.951 INFO:teuthology.orchestra.run.smithi073.stdout: ceph x86_64 2:18.2.2-1260.g5403a021.el9 ceph 6.5 k 2024-04-27T03:51:49.951 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-04-27T03:51:49.951 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mds x86_64 2:18.2.2-1260.g5403a021.el9 ceph 2.1 M 2024-04-27T03:51:49.951 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr x86_64 2:18.2.2-1260.g5403a021.el9 ceph 1.5 M 2024-04-27T03:51:49.951 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1260.g5403a021.el9 ceph-noarch 246 k 2024-04-27T03:51:49.951 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mon x86_64 2:18.2.2-1260.g5403a021.el9 ceph 4.7 M 2024-04-27T03:51:49.951 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-osd x86_64 2:18.2.2-1260.g5403a021.el9 ceph 17 M 2024-04-27T03:51:49.951 INFO:teuthology.orchestra.run.smithi073.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-27T03:51:49.951 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-27T03:51:49.951 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-27T03:51:49.951 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-27T03:51:49.952 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-27T03:51:49.952 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-27T03:51:49.952 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-27T03:51:49.952 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-collections 2024-04-27T03:51:49.952 INFO:teuthology.orchestra.run.smithi073.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-27T03:51:49.952 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-27T03:51:49.952 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-27T03:51:49.952 INFO:teuthology.orchestra.run.smithi073.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-27T03:51:49.952 INFO:teuthology.orchestra.run.smithi073.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-27T03:51:49.952 INFO:teuthology.orchestra.run.smithi073.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-27T03:51:49.952 INFO:teuthology.orchestra.run.smithi073.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-27T03:51:49.952 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-27T03:51:49.952 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-27T03:51:49.952 INFO:teuthology.orchestra.run.smithi073.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-27T03:51:49.953 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-27T03:51:49.953 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-27T03:51:49.953 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-27T03:51:49.953 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-27T03:51:49.953 INFO:teuthology.orchestra.run.smithi073.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-27T03:51:49.953 INFO:teuthology.orchestra.run.smithi073.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-27T03:51:49.953 INFO:teuthology.orchestra.run.smithi073.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-27T03:51:49.953 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-27T03:51:49.953 INFO:teuthology.orchestra.run.smithi073.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-27T03:51:49.953 INFO:teuthology.orchestra.run.smithi073.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-27T03:51:49.953 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:51:49.953 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:51:49.953 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:51:49.954 INFO:teuthology.orchestra.run.smithi073.stdout:Install 33 Packages 2024-04-27T03:51:49.954 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:51:49.954 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 29 M 2024-04-27T03:51:49.954 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 104 M 2024-04-27T03:51:49.954 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-04-27T03:51:50.815 INFO:teuthology.orchestra.run.smithi073.stdout:(1/33): ceph-18.2.2-1260.g5403a021.el9.x86_64.r 30 kB/s | 6.5 kB 00:00 2024-04-27T03:51:51.132 INFO:teuthology.orchestra.run.smithi073.stdout:(2/33): ceph-mgr-18.2.2-1260.g5403a021.el9.x86_ 2.7 MB/s | 1.5 MB 00:00 2024-04-27T03:51:51.307 INFO:teuthology.orchestra.run.smithi073.stdout:(3/33): ceph-mon-18.2.2-1260.g5403a021.el9.x86_ 9.5 MB/s | 4.7 MB 00:00 2024-04-27T03:51:51.374 INFO:teuthology.orchestra.run.smithi073.stdout:(4/33): ceph-mgr-modules-core-18.2.2-1260.g5403 3.6 MB/s | 246 kB 00:00 2024-04-27T03:51:52.033 INFO:teuthology.orchestra.run.smithi073.stdout:(5/33): ceph-osd-18.2.2-1260.g5403a021.el9.x86_ 18 MB/s | 17 MB 00:00 2024-04-27T03:51:52.191 INFO:teuthology.orchestra.run.smithi073.stdout:(6/33): ceph-mds-18.2.2-1260.g5403a021.el9.x86_ 1.3 MB/s | 2.1 MB 00:01 2024-04-27T03:51:52.350 INFO:teuthology.orchestra.run.smithi073.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 259 kB/s | 253 kB 00:00 2024-04-27T03:51:52.942 INFO:teuthology.orchestra.run.smithi073.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 142 kB/s | 106 kB 00:00 2024-04-27T03:51:53.159 INFO:teuthology.orchestra.run.smithi073.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 167 kB/s | 135 kB 00:00 2024-04-27T03:51:53.518 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1260.g5403a021.el9.x86_64 3/4 2024-04-27T03:51:53.598 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-test-2:18.2.2-1260.g5403a021.el9.x86_64 4/4 2024-04-27T03:51:53.701 INFO:teuthology.orchestra.run.smithi073.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 165 kB/s | 125 kB 00:00 2024-04-27T03:51:53.843 INFO:teuthology.orchestra.run.smithi073.stdout:(11/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-04-27T03:51:53.893 INFO:teuthology.orchestra.run.smithi073.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 698 kB/s | 35 kB 00:00 2024-04-27T03:51:53.944 INFO:teuthology.orchestra.run.smithi073.stdout:(13/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-27T03:51:53.994 INFO:teuthology.orchestra.run.smithi073.stdout:(14/33): python3-toml-0.10.2-6.el9.noarch.rpm 833 kB/s | 42 kB 00:00 2024-04-27T03:51:54.019 INFO:teuthology.orchestra.run.smithi073.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 250 kB/s | 215 kB 00:00 2024-04-27T03:51:54.211 INFO:teuthology.orchestra.run.smithi073.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 200 kB/s | 43 kB 00:00 2024-04-27T03:51:54.261 INFO:teuthology.orchestra.run.smithi073.stdout:(17/33): python3-cryptography-36.0.1-4.el9.x86_ 563 kB/s | 1.2 MB 00:02 2024-04-27T03:51:54.286 INFO:teuthology.orchestra.run.smithi073.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 643 kB/s | 172 kB 00:00 2024-04-27T03:51:54.345 INFO:teuthology.orchestra.run.smithi073.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-04-27T03:51:54.379 INFO:teuthology.orchestra.run.smithi073.stdout:(20/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-04-27T03:51:54.404 INFO:teuthology.orchestra.run.smithi073.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 396 kB/s | 23 kB 00:00 2024-04-27T03:51:54.429 INFO:teuthology.orchestra.run.smithi073.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-04-27T03:51:54.454 INFO:teuthology.orchestra.run.smithi073.stdout:(23/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 55 kB/s | 11 kB 00:00 2024-04-27T03:51:54.479 INFO:teuthology.orchestra.run.smithi073.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-04-27T03:51:54.505 INFO:teuthology.orchestra.run.smithi073.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-04-27T03:51:54.564 INFO:teuthology.orchestra.run.smithi073.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 282 kB/s | 16 kB 00:00 2024-04-27T03:51:54.589 INFO:teuthology.orchestra.run.smithi073.stdout:(27/33): python3-more-itertools-8.12.0-2.el9.no 587 kB/s | 79 kB 00:00 2024-04-27T03:51:54.614 INFO:teuthology.orchestra.run.smithi073.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-04-27T03:51:54.639 INFO:teuthology.orchestra.run.smithi073.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-04-27T03:51:54.664 INFO:teuthology.orchestra.run.smithi073.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 475 kB/s | 36 kB 00:00 2024-04-27T03:51:54.698 INFO:teuthology.orchestra.run.smithi073.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-27T03:51:54.723 INFO:teuthology.orchestra.run.smithi073.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-04-27T03:51:54.757 INFO:teuthology.orchestra.run.smithi073.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-04-27T03:51:54.763 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:51:54.763 INFO:teuthology.orchestra.run.smithi073.stdout:Total 6.1 MB/s | 29 MB 00:04 2024-04-27T03:51:54.879 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-04-27T03:51:54.916 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-04-27T03:51:54.916 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-04-27T03:51:55.196 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-04-27T03:51:55.197 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-04-27T03:51:55.607 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-test-2:18.2.2-1260.g5403a021.el9.x86_64 4/4 2024-04-27T03:51:55.607 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-test-2:18.2.2-1260.g5403a021.el9.x86_64 1/4 2024-04-27T03:51:55.607 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephsqlite-2:18.2.2-1260.g5403a021.el9.x86_64 2/4 2024-04-27T03:51:55.607 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-27T03:51:55.894 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-27T03:51:55.894 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:51:55.895 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-27T03:51:55.895 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:55.895 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:51:55.895 INFO:teuthology.orchestra.run.smithi046.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-27T03:51:55.895 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-27T03:51:55.895 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:51:55.895 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:51:56.156 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph 2024-04-27T03:51:56.355 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-04-27T03:51:56.426 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-27T03:51:56.459 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-27T03:51:56.492 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-27T03:51:56.585 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-27T03:51:56.663 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-27T03:51:56.686 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:39 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:51:56.711 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-27T03:51:56.746 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-27T03:51:56.776 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-27T03:51:56.810 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-27T03:51:56.814 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:51:56.817 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:51:56.817 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-04-27T03:51:56.817 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:51:56.817 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-27T03:51:56.817 INFO:teuthology.orchestra.run.smithi046.stdout: ceph x86_64 2:18.2.2-1260.g5403a021.el9 ceph 6.5 k 2024-04-27T03:51:56.817 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-04-27T03:51:56.818 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds x86_64 2:18.2.2-1260.g5403a021.el9 ceph 2.1 M 2024-04-27T03:51:56.818 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr x86_64 2:18.2.2-1260.g5403a021.el9 ceph 1.5 M 2024-04-27T03:51:56.818 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1260.g5403a021.el9 ceph-noarch 246 k 2024-04-27T03:51:56.818 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon x86_64 2:18.2.2-1260.g5403a021.el9 ceph 4.7 M 2024-04-27T03:51:56.818 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd x86_64 2:18.2.2-1260.g5403a021.el9 ceph 17 M 2024-04-27T03:51:56.818 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-27T03:51:56.818 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-27T03:51:56.818 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-27T03:51:56.818 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-27T03:51:56.818 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-27T03:51:56.818 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-27T03:51:56.818 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-27T03:51:56.819 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections 2024-04-27T03:51:56.819 INFO:teuthology.orchestra.run.smithi046.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-27T03:51:56.819 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-27T03:51:56.819 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-27T03:51:56.819 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-27T03:51:56.819 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-27T03:51:56.819 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-27T03:51:56.819 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-27T03:51:56.819 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-27T03:51:56.819 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-27T03:51:56.819 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-27T03:51:56.819 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-27T03:51:56.820 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-27T03:51:56.820 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-27T03:51:56.820 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-27T03:51:56.820 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-27T03:51:56.820 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-27T03:51:56.820 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-27T03:51:56.820 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-27T03:51:56.820 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-27T03:51:56.820 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-27T03:51:56.820 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:51:56.821 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:51:56.821 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:51:56.821 INFO:teuthology.orchestra.run.smithi046.stdout:Install 33 Packages 2024-04-27T03:51:56.821 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:51:56.822 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 29 M 2024-04-27T03:51:56.822 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 104 M 2024-04-27T03:51:56.822 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-27T03:51:56.843 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-27T03:51:56.884 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-27T03:51:56.919 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-27T03:51:56.953 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-27T03:51:56.992 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-27T03:51:57.066 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-27T03:51:57.213 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-27T03:51:57.292 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-27T03:51:57.345 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-27T03:51:57.391 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-27T03:51:57.448 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-27T03:51:57.504 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-27T03:51:57.693 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-27T03:51:57.745 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-27T03:51:57.763 INFO:teuthology.orchestra.run.smithi046.stdout:(1/33): ceph-18.2.2-1260.g5403a021.el9.x86_64.r 28 kB/s | 6.5 kB 00:00 2024-04-27T03:51:57.825 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-27T03:51:57.905 INFO:teuthology.orchestra.run.smithi046.stdout:(2/33): ceph-mds-18.2.2-1260.g5403a021.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-04-27T03:51:57.914 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-27T03:51:57.953 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-27T03:51:58.071 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-27T03:51:58.236 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1260.g5403a021.el9. 28/33 2024-04-27T03:51:58.263 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-2:18.2.2-1260.g5403a021.el9.x86_64 29/33 2024-04-27T03:51:58.288 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1260.g5403a021.el9.x86_64 29/33 2024-04-27T03:51:58.288 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T03:51:58.288 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-27T03:51:58.288 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-27T03:51:58.288 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-27T03:51:58.288 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:51:58.514 INFO:teuthology.orchestra.run.smithi046.stdout:(3/33): ceph-mgr-18.2.2-1260.g5403a021.el9.x86_ 1.5 MB/s | 1.5 MB 00:00 2024-04-27T03:51:58.730 INFO:teuthology.orchestra.run.smithi046.stdout:(4/33): ceph-osd-18.2.2-1260.g5403a021.el9.x86_ 20 MB/s | 17 MB 00:00 2024-04-27T03:51:58.755 INFO:teuthology.orchestra.run.smithi046.stdout:(5/33): ceph-mgr-modules-core-18.2.2-1260.g5403 1.0 MB/s | 246 kB 00:00 2024-04-27T03:51:59.106 INFO:teuthology.orchestra.run.smithi046.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 675 kB/s | 253 kB 00:00 2024-04-27T03:51:59.198 INFO:teuthology.orchestra.run.smithi046.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-04-27T03:51:59.231 INFO:teuthology.orchestra.run.smithi046.stdout:(8/33): python3-cryptography-36.0.1-4.el9.x86_6 2.6 MB/s | 1.2 MB 00:00 2024-04-27T03:51:59.290 INFO:teuthology.orchestra.run.smithi046.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 1.4 MB/s | 135 kB 00:00 2024-04-27T03:51:59.315 INFO:teuthology.orchestra.run.smithi046.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 1.5 MB/s | 125 kB 00:00 2024-04-27T03:51:59.357 INFO:teuthology.orchestra.run.smithi046.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-04-27T03:51:59.441 INFO:teuthology.orchestra.run.smithi046.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 417 kB/s | 35 kB 00:00 2024-04-27T03:51:59.466 INFO:teuthology.orchestra.run.smithi046.stdout:(13/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-27T03:51:59.508 INFO:teuthology.orchestra.run.smithi046.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 761 kB/s | 51 kB 00:00 2024-04-27T03:51:59.533 INFO:teuthology.orchestra.run.smithi046.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 622 kB/s | 42 kB 00:00 2024-04-27T03:51:59.722 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-osd-2:18.2.2-1260.g5403a021.el9.x86_64 30/33 2024-04-27T03:51:59.749 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-osd-2:18.2.2-1260.g5403a021.el9.x86_64 30/33 2024-04-27T03:51:59.749 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T03:51:59.749 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-27T03:51:59.749 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-27T03:51:59.749 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-27T03:51:59.749 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:51:59.842 INFO:teuthology.orchestra.run.smithi046.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 130 kB/s | 43 kB 00:00 2024-04-27T03:51:59.942 INFO:teuthology.orchestra.run.smithi046.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 421 kB/s | 172 kB 00:00 2024-04-27T03:52:00.034 INFO:teuthology.orchestra.run.smithi046.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 117 kB/s | 11 kB 00:00 2024-04-27T03:52:00.084 INFO:teuthology.orchestra.run.smithi046.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-04-27T03:52:00.118 INFO:teuthology.orchestra.run.smithi046.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-04-27T03:52:00.193 INFO:teuthology.orchestra.run.smithi046.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-04-27T03:52:00.207 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mon-2:18.2.2-1260.g5403a021.el9.x86_64 31/33 2024-04-27T03:52:00.210 INFO:teuthology.orchestra.run.smithi046.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 212 kB/s | 19 kB 00:00 2024-04-27T03:52:00.231 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mon-2:18.2.2-1260.g5403a021.el9.x86_64 31/33 2024-04-27T03:52:00.231 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T03:52:00.231 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-27T03:52:00.231 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-27T03:52:00.231 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-27T03:52:00.231 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:00.285 INFO:teuthology.orchestra.run.smithi046.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-04-27T03:52:00.302 INFO:teuthology.orchestra.run.smithi046.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 505 kB/s | 46 kB 00:00 2024-04-27T03:52:00.378 INFO:teuthology.orchestra.run.smithi046.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 858 kB/s | 79 kB 00:00 2024-04-27T03:52:00.449 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mds-2:18.2.2-1260.g5403a021.el9.x86_64 32/33 2024-04-27T03:52:00.469 INFO:teuthology.orchestra.run.smithi046.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-04-27T03:52:00.472 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mds-2:18.2.2-1260.g5403a021.el9.x86_64 32/33 2024-04-27T03:52:00.472 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T03:52:00.472 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-27T03:52:00.472 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-27T03:52:00.472 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-27T03:52:00.473 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:00.495 INFO:teuthology.orchestra.run.smithi046.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 140 kB/s | 16 kB 00:00 2024-04-27T03:52:00.544 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-2:18.2.2-1260.g5403a021.el9.x86_64 33/33 2024-04-27T03:52:00.603 INFO:teuthology.orchestra.run.smithi046.stdout:(28/33): ceph-mon-18.2.2-1260.g5403a021.el9.x86 1.6 MB/s | 4.7 MB 00:02 2024-04-27T03:52:00.629 INFO:teuthology.orchestra.run.smithi046.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 566 kB/s | 90 kB 00:00 2024-04-27T03:52:00.654 INFO:teuthology.orchestra.run.smithi046.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 225 kB/s | 36 kB 00:00 2024-04-27T03:52:00.746 INFO:teuthology.orchestra.run.smithi046.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-04-27T03:52:00.771 INFO:teuthology.orchestra.run.smithi046.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2024-04-27T03:52:01.054 INFO:teuthology.orchestra.run.smithi046.stdout:(33/33): python3-webob-1.8.7-6.el9.noarch.rpm 509 kB/s | 230 kB 00:00 2024-04-27T03:52:01.058 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:01.058 INFO:teuthology.orchestra.run.smithi046.stdout:Total 6.9 MB/s | 29 MB 00:04 2024-04-27T03:52:01.181 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-27T03:52:01.222 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-27T03:52:01.223 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-27T03:52:01.519 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-27T03:52:01.519 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-27T03:52:02.289 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-2:18.2.2-1260.g5403a021.el9.x86_64 33/33 2024-04-27T03:52:02.289 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-2:18.2.2-1260.g5403a021.el9.x86_64 1/33 2024-04-27T03:52:02.289 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mds-2:18.2.2-1260.g5403a021.el9.x86_64 2/33 2024-04-27T03:52:02.290 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-2:18.2.2-1260.g5403a021.el9.x86_64 3/33 2024-04-27T03:52:02.290 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mon-2:18.2.2-1260.g5403a021.el9.x86_64 4/33 2024-04-27T03:52:02.290 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-osd-2:18.2.2-1260.g5403a021.el9.x86_64 5/33 2024-04-27T03:52:02.290 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1260.g5403a021.el9. 6/33 2024-04-27T03:52:02.290 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-27T03:52:02.290 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-27T03:52:02.291 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-27T03:52:02.291 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-27T03:52:02.291 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-27T03:52:02.291 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-27T03:52:02.291 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-27T03:52:02.291 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-27T03:52:02.292 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-27T03:52:02.292 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-27T03:52:02.292 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-27T03:52:02.292 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-27T03:52:02.292 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-27T03:52:02.292 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-27T03:52:02.293 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-27T03:52:02.293 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-27T03:52:02.293 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-27T03:52:02.293 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-27T03:52:02.293 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-27T03:52:02.293 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-27T03:52:02.293 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-27T03:52:02.293 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-27T03:52:02.293 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-27T03:52:02.293 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-27T03:52:02.293 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-27T03:52:02.294 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-27T03:52:02.678 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-27T03:52:02.741 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-27T03:52:02.773 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-27T03:52:02.806 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-27T03:52:02.901 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-27T03:52:02.987 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-27T03:52:03.034 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-27T03:52:03.069 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-27T03:52:03.098 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-27T03:52:03.133 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-27T03:52:03.164 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-27T03:52:03.164 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:03.164 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-04-27T03:52:03.164 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:52:03.164 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mds-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:52:03.164 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:52:03.164 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-modules-core-2:18.2.2-1260.g5403a021.el9.noarch 2024-04-27T03:52:03.165 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mon-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:52:03.165 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-osd-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:52:03.165 INFO:teuthology.orchestra.run.smithi073.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-27T03:52:03.165 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-27T03:52:03.165 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-27T03:52:03.165 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-27T03:52:03.165 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-27T03:52:03.165 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-27T03:52:03.165 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-27T03:52:03.165 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-27T03:52:03.165 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-27T03:52:03.165 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-27T03:52:03.165 INFO:teuthology.orchestra.run.smithi073.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-27T03:52:03.166 INFO:teuthology.orchestra.run.smithi073.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-27T03:52:03.166 INFO:teuthology.orchestra.run.smithi073.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-27T03:52:03.166 INFO:teuthology.orchestra.run.smithi073.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-27T03:52:03.166 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-27T03:52:03.166 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ply-3.11-14.el9.noarch 2024-04-27T03:52:03.166 INFO:teuthology.orchestra.run.smithi073.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-27T03:52:03.166 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-27T03:52:03.166 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-27T03:52:03.166 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-27T03:52:03.166 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-27T03:52:03.166 INFO:teuthology.orchestra.run.smithi073.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-27T03:52:03.166 INFO:teuthology.orchestra.run.smithi073.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-27T03:52:03.166 INFO:teuthology.orchestra.run.smithi073.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-27T03:52:03.167 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-27T03:52:03.167 INFO:teuthology.orchestra.run.smithi073.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-27T03:52:03.167 INFO:teuthology.orchestra.run.smithi073.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-27T03:52:03.167 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:03.167 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:52:03.167 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-27T03:52:03.224 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-27T03:52:03.267 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-27T03:52:03.301 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-27T03:52:03.340 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-27T03:52:03.415 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-27T03:52:03.520 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-base 2024-04-27T03:52:03.558 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-27T03:52:03.642 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-27T03:52:03.701 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-27T03:52:03.739 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-27T03:52:03.797 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-27T03:52:03.861 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-27T03:52:04.036 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:51 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:52:04.056 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-27T03:52:04.101 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-27T03:52:04.114 INFO:teuthology.orchestra.run.smithi073.stdout:Package ceph-base-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:52:04.155 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:52:04.156 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-04-27T03:52:04.156 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:52:04.182 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-27T03:52:04.224 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install cephadm 2024-04-27T03:52:04.271 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-27T03:52:04.309 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-27T03:52:04.431 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-27T03:52:04.598 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1260.g5403a021.el9. 28/33 2024-04-27T03:52:04.627 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-2:18.2.2-1260.g5403a021.el9.x86_64 29/33 2024-04-27T03:52:04.654 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1260.g5403a021.el9.x86_64 29/33 2024-04-27T03:52:04.654 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T03:52:04.654 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-27T03:52:04.654 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-27T03:52:04.654 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-27T03:52:04.654 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:04.733 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:51 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:52:04.853 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:52:04.854 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:52:04.854 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-04-27T03:52:04.854 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:52:04.854 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-04-27T03:52:04.854 INFO:teuthology.orchestra.run.smithi073.stdout: cephadm noarch 2:18.2.2-1260.g5403a021.el9 ceph-noarch 224 k 2024-04-27T03:52:04.854 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:04.855 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:52:04.855 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:52:04.855 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-04-27T03:52:04.855 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:04.855 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 224 k 2024-04-27T03:52:04.855 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 216 k 2024-04-27T03:52:04.855 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-04-27T03:52:05.196 INFO:teuthology.orchestra.run.smithi073.stdout:cephadm-18.2.2-1260.g5403a021.el9.noarch.rpm 658 kB/s | 224 kB 00:00 2024-04-27T03:52:05.197 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:05.197 INFO:teuthology.orchestra.run.smithi073.stdout:Total 655 kB/s | 224 kB 00:00 2024-04-27T03:52:05.197 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-04-27T03:52:05.202 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-04-27T03:52:05.202 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-04-27T03:52:05.205 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-04-27T03:52:05.206 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-04-27T03:52:05.274 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-04-27T03:52:05.734 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: cephadm-2:18.2.2-1260.g5403a021.el9.noarch 1/1 2024-04-27T03:52:05.816 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : cephadm-2:18.2.2-1260.g5403a021.el9.noarch 1/1 2024-04-27T03:52:06.082 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: cephadm-2:18.2.2-1260.g5403a021.el9.noarch 1/1 2024-04-27T03:52:06.146 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-osd-2:18.2.2-1260.g5403a021.el9.x86_64 30/33 2024-04-27T03:52:06.174 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-osd-2:18.2.2-1260.g5403a021.el9.x86_64 30/33 2024-04-27T03:52:06.175 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T03:52:06.175 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-27T03:52:06.175 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-27T03:52:06.175 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-27T03:52:06.175 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:06.328 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : cephadm-2:18.2.2-1260.g5403a021.el9.noarch 1/1 2024-04-27T03:52:06.328 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:06.328 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-04-27T03:52:06.328 INFO:teuthology.orchestra.run.smithi073.stdout: cephadm-2:18.2.2-1260.g5403a021.el9.noarch 2024-04-27T03:52:06.328 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:06.328 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:52:06.520 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-immutable-object-cache 2024-04-27T03:52:06.646 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mon-2:18.2.2-1260.g5403a021.el9.x86_64 31/33 2024-04-27T03:52:06.672 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mon-2:18.2.2-1260.g5403a021.el9.x86_64 31/33 2024-04-27T03:52:06.672 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T03:52:06.672 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-27T03:52:06.672 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-27T03:52:06.672 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-27T03:52:06.672 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:06.880 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mds-2:18.2.2-1260.g5403a021.el9.x86_64 32/33 2024-04-27T03:52:06.905 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mds-2:18.2.2-1260.g5403a021.el9.x86_64 32/33 2024-04-27T03:52:06.905 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T03:52:06.905 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-27T03:52:06.905 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-27T03:52:06.905 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-27T03:52:06.905 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:06.969 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-2:18.2.2-1260.g5403a021.el9.x86_64 33/33 2024-04-27T03:52:07.035 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:54 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:52:07.159 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:52:07.159 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:52:07.159 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2024-04-27T03:52:07.159 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:52:07.159 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-04-27T03:52:07.159 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1260.g5403a021.el9 ceph 142 k 2024-04-27T03:52:07.160 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:07.160 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:52:07.160 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:52:07.160 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-04-27T03:52:07.160 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:07.160 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 142 k 2024-04-27T03:52:07.161 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 395 k 2024-04-27T03:52:07.161 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-04-27T03:52:07.502 INFO:teuthology.orchestra.run.smithi073.stdout:ceph-immutable-object-cache-18.2.2-1260.g5403a0 417 kB/s | 142 kB 00:00 2024-04-27T03:52:07.503 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:07.504 INFO:teuthology.orchestra.run.smithi073.stdout:Total 415 kB/s | 142 kB 00:00 2024-04-27T03:52:07.504 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-04-27T03:52:07.511 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-04-27T03:52:07.511 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-04-27T03:52:07.568 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-04-27T03:52:07.568 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-04-27T03:52:07.721 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-04-27T03:52:07.803 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1260.g5403a021. 1/1 2024-04-27T03:52:07.828 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1260.g5403a021. 1/1 2024-04-27T03:52:07.828 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T03:52:07.828 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-27T03:52:07.828 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-2:18.2.2-1260.g5403a021.el9.x86_64 33/33 2024-04-27T03:52:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-2:18.2.2-1260.g5403a021.el9.x86_64 1/33 2024-04-27T03:52:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mds-2:18.2.2-1260.g5403a021.el9.x86_64 2/33 2024-04-27T03:52:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-2:18.2.2-1260.g5403a021.el9.x86_64 3/33 2024-04-27T03:52:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mon-2:18.2.2-1260.g5403a021.el9.x86_64 4/33 2024-04-27T03:52:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-osd-2:18.2.2-1260.g5403a021.el9.x86_64 5/33 2024-04-27T03:52:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1260.g5403a021.el9. 6/33 2024-04-27T03:52:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-27T03:52:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-27T03:52:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-27T03:52:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-27T03:52:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-27T03:52:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-27T03:52:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-27T03:52:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-27T03:52:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-27T03:52:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-27T03:52:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-27T03:52:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-27T03:52:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-27T03:52:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-27T03:52:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-27T03:52:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-27T03:52:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-27T03:52:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-27T03:52:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-27T03:52:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-27T03:52:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-27T03:52:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-27T03:52:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-27T03:52:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-27T03:52:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-27T03:52:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-27T03:52:08.676 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1260.g5403a021. 1/1 2024-04-27T03:52:08.677 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:08.677 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-04-27T03:52:08.677 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-immutable-object-cache-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:52:08.677 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:08.677 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:52:08.899 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr 2024-04-27T03:52:09.408 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:56 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:52:09.423 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-27T03:52:09.423 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:09.423 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-27T03:52:09.423 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:52:09.423 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:52:09.423 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:52:09.424 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core-2:18.2.2-1260.g5403a021.el9.noarch 2024-04-27T03:52:09.424 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:52:09.424 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:52:09.424 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-27T03:52:09.424 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-27T03:52:09.424 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-27T03:52:09.424 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-27T03:52:09.424 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-27T03:52:09.425 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-27T03:52:09.425 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-27T03:52:09.425 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-27T03:52:09.425 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-27T03:52:09.425 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-27T03:52:09.425 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-27T03:52:09.425 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-27T03:52:09.425 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-27T03:52:09.425 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-27T03:52:09.426 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-27T03:52:09.426 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply-3.11-14.el9.noarch 2024-04-27T03:52:09.426 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-27T03:52:09.426 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-27T03:52:09.426 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-27T03:52:09.426 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-27T03:52:09.426 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-27T03:52:09.426 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-27T03:52:09.426 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-27T03:52:09.427 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-27T03:52:09.427 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-27T03:52:09.427 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-27T03:52:09.427 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-27T03:52:09.427 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:09.427 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:09.487 INFO:teuthology.orchestra.run.smithi073.stdout:Package ceph-mgr-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:52:09.530 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:52:09.531 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-04-27T03:52:09.532 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:52:09.595 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-dashboard 2024-04-27T03:52:09.761 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-base 2024-04-27T03:52:10.106 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:57 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:52:10.234 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:52:10.236 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:52:10.236 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-04-27T03:52:10.236 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:52:10.236 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-04-27T03:52:10.236 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1260.g5403a021.el9 ceph-noarch 3.5 M 2024-04-27T03:52:10.236 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-04-27T03:52:10.236 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1260.g5403a021.el9 ceph-noarch 24 k 2024-04-27T03:52:10.236 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1260.g5403a021.el9 ceph-noarch 15 k 2024-04-27T03:52:10.236 INFO:teuthology.orchestra.run.smithi073.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-27T03:52:10.237 INFO:teuthology.orchestra.run.smithi073.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-27T03:52:10.237 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:10.237 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:52:10.237 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:52:10.237 INFO:teuthology.orchestra.run.smithi073.stdout:Install 5 Packages 2024-04-27T03:52:10.237 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:10.237 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 3.8 M 2024-04-27T03:52:10.237 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 87 M 2024-04-27T03:52:10.238 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-04-27T03:52:10.297 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:53 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:10.379 INFO:teuthology.orchestra.run.smithi046.stdout:Package ceph-base-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:52:10.422 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:10.423 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-04-27T03:52:10.423 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:10.492 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install cephadm 2024-04-27T03:52:10.514 INFO:teuthology.orchestra.run.smithi073.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1260.g540 125 kB/s | 24 kB 00:00 2024-04-27T03:52:10.614 INFO:teuthology.orchestra.run.smithi073.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1260.g5403 52 kB/s | 15 kB 00:00 2024-04-27T03:52:10.648 INFO:teuthology.orchestra.run.smithi073.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 230 kB/s | 31 kB 00:00 2024-04-27T03:52:10.723 INFO:teuthology.orchestra.run.smithi073.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-04-27T03:52:10.792 INFO:teuthology.orchestra.run.smithi073.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1260.g5403a021 7.5 MB/s | 3.5 MB 00:00 2024-04-27T03:52:10.797 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:10.797 INFO:teuthology.orchestra.run.smithi073.stdout:Total 6.8 MB/s | 3.8 MB 00:00 2024-04-27T03:52:10.797 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-04-27T03:52:10.809 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-04-27T03:52:10.809 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-04-27T03:52:10.875 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-04-27T03:52:10.875 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-04-27T03:52:11.022 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:54 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:11.124 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-04-27T03:52:11.148 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:11.148 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:11.149 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-04-27T03:52:11.149 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:11.149 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-27T03:52:11.149 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm noarch 2:18.2.2-1260.g5403a021.el9 ceph-noarch 224 k 2024-04-27T03:52:11.149 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:11.149 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:52:11.149 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:11.149 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-04-27T03:52:11.149 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:11.149 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 224 k 2024-04-27T03:52:11.149 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 216 k 2024-04-27T03:52:11.149 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-27T03:52:11.196 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-04-27T03:52:11.228 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-04-27T03:52:11.258 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1260.g5403a021.el9.n 3/5 2024-04-27T03:52:11.411 INFO:teuthology.orchestra.run.smithi046.stdout:cephadm-18.2.2-1260.g5403a021.el9.noarch.rpm 858 kB/s | 224 kB 00:00 2024-04-27T03:52:11.412 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:11.412 INFO:teuthology.orchestra.run.smithi046.stdout:Total 853 kB/s | 224 kB 00:00 2024-04-27T03:52:11.412 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-27T03:52:11.418 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-27T03:52:11.418 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-27T03:52:11.421 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-27T03:52:11.421 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-27T03:52:11.494 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-27T03:52:11.949 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:18.2.2-1260.g5403a021.el9.noarch 1/1 2024-04-27T03:52:12.041 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephadm-2:18.2.2-1260.g5403a021.el9.noarch 1/1 2024-04-27T03:52:12.304 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1260.g5403a021.el9. 4/5 2024-04-27T03:52:12.344 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1260.g5403a021.el9.noarc 5/5 2024-04-27T03:52:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:18.2.2-1260.g5403a021.el9.noarch 1/1 2024-04-27T03:52:12.614 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephadm-2:18.2.2-1260.g5403a021.el9.noarch 1/1 2024-04-27T03:52:12.614 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:12.614 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-27T03:52:12.614 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm-2:18.2.2-1260.g5403a021.el9.noarch 2024-04-27T03:52:12.614 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:12.614 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:12.806 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-immutable-object-cache 2024-04-27T03:52:13.353 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:56 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:13.444 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1260.g5403a021.el9.noarc 5/5 2024-04-27T03:52:13.444 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1260.g5403a021.el9. 1/5 2024-04-27T03:52:13.444 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1260.g5403a021.el9.noarc 2/5 2024-04-27T03:52:13.444 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1260.g5403a021.el9.n 3/5 2024-04-27T03:52:13.444 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-04-27T03:52:13.469 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:13.470 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:13.470 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-04-27T03:52:13.470 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:13.470 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-27T03:52:13.470 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1260.g5403a021.el9 ceph 142 k 2024-04-27T03:52:13.470 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:13.470 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:52:13.470 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:13.470 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-04-27T03:52:13.470 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:13.471 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 142 k 2024-04-27T03:52:13.471 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 395 k 2024-04-27T03:52:13.471 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-27T03:52:13.692 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-04-27T03:52:13.692 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:13.692 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-04-27T03:52:13.692 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-grafana-dashboards-2:18.2.2-1260.g5403a021.el9.noarch 2024-04-27T03:52:13.692 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-dashboard-2:18.2.2-1260.g5403a021.el9.noarch 2024-04-27T03:52:13.692 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-prometheus-alerts-2:18.2.2-1260.g5403a021.el9.noarch 2024-04-27T03:52:13.692 INFO:teuthology.orchestra.run.smithi073.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-27T03:52:13.692 INFO:teuthology.orchestra.run.smithi073.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-27T03:52:13.692 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:13.693 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:52:13.792 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-immutable-object-cache-18.2.2-1260.g5403a0 444 kB/s | 142 kB 00:00 2024-04-27T03:52:13.792 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:13.792 INFO:teuthology.orchestra.run.smithi046.stdout:Total 442 kB/s | 142 kB 00:00 2024-04-27T03:52:13.793 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-27T03:52:13.801 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-27T03:52:13.801 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-27T03:52:13.862 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-27T03:52:13.862 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-27T03:52:13.893 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-27T03:52:14.018 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-27T03:52:14.109 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1260.g5403a021. 1/1 2024-04-27T03:52:14.136 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1260.g5403a021. 1/1 2024-04-27T03:52:14.136 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T03:52:14.136 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-27T03:52:14.136 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:14.401 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:01 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:52:14.522 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:52:14.523 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================= 2024-04-27T03:52:14.523 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-04-27T03:52:14.523 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================= 2024-04-27T03:52:14.524 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-04-27T03:52:14.524 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1260.g5403a021.el9 ceph-noarch 7.4 M 2024-04-27T03:52:14.524 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-04-27T03:52:14.524 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-27T03:52:14.524 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-27T03:52:14.524 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-27T03:52:14.524 INFO:teuthology.orchestra.run.smithi073.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-27T03:52:14.524 INFO:teuthology.orchestra.run.smithi073.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-27T03:52:14.524 INFO:teuthology.orchestra.run.smithi073.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-27T03:52:14.524 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-27T03:52:14.524 INFO:teuthology.orchestra.run.smithi073.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-27T03:52:14.524 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-27T03:52:14.524 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-27T03:52:14.524 INFO:teuthology.orchestra.run.smithi073.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-27T03:52:14.525 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:14.525 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:52:14.525 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================= 2024-04-27T03:52:14.525 INFO:teuthology.orchestra.run.smithi073.stdout:Install 12 Packages 2024-04-27T03:52:14.525 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:14.525 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 38 M 2024-04-27T03:52:14.525 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 209 M 2024-04-27T03:52:14.526 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-04-27T03:52:15.059 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1260.g5403a021. 1/1 2024-04-27T03:52:15.059 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:15.059 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-27T03:52:15.059 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:52:15.059 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:15.059 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:15.288 INFO:teuthology.orchestra.run.smithi073.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 953 kB/s | 195 kB 00:00 2024-04-27T03:52:15.291 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr 2024-04-27T03:52:15.422 INFO:teuthology.orchestra.run.smithi073.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-04-27T03:52:15.824 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:58 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:15.831 INFO:teuthology.orchestra.run.smithi073.stdout:(3/12): ceph-mgr-diskprediction-local-18.2.2-12 9.9 MB/s | 7.4 MB 00:00 2024-04-27T03:52:15.902 INFO:teuthology.orchestra.run.smithi073.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 53 kB/s | 33 kB 00:00 2024-04-27T03:52:15.906 INFO:teuthology.orchestra.run.smithi046.stdout:Package ceph-mgr-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:52:15.950 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:15.951 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-04-27T03:52:15.951 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:16.016 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-dashboard 2024-04-27T03:52:16.373 INFO:teuthology.orchestra.run.smithi073.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 34 kB/s | 18 kB 00:00 2024-04-27T03:52:16.523 INFO:teuthology.orchestra.run.smithi073.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 57 kB/s | 35 kB 00:00 2024-04-27T03:52:16.548 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:59 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:16.677 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:16.677 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:16.678 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-04-27T03:52:16.678 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:16.678 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-27T03:52:16.678 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1260.g5403a021.el9 ceph-noarch 3.5 M 2024-04-27T03:52:16.678 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-04-27T03:52:16.678 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1260.g5403a021.el9 ceph-noarch 24 k 2024-04-27T03:52:16.678 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1260.g5403a021.el9 ceph-noarch 15 k 2024-04-27T03:52:16.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-27T03:52:16.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-27T03:52:16.678 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:16.678 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:52:16.678 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:16.679 INFO:teuthology.orchestra.run.smithi046.stdout:Install 5 Packages 2024-04-27T03:52:16.679 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:16.679 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 3.8 M 2024-04-27T03:52:16.679 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 87 M 2024-04-27T03:52:16.679 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-27T03:52:17.237 INFO:teuthology.orchestra.run.smithi046.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1260.g540 124 kB/s | 24 kB 00:00 2024-04-27T03:52:17.262 INFO:teuthology.orchestra.run.smithi046.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1260.g5403 69 kB/s | 15 kB 00:00 2024-04-27T03:52:17.504 INFO:teuthology.orchestra.run.smithi046.stdout:(3/5): ceph-mgr-dashboard-18.2.2-1260.g5403a021 7.6 MB/s | 3.5 MB 00:00 2024-04-27T03:52:17.523 INFO:teuthology.orchestra.run.smithi073.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 244 kB/s | 244 kB 00:00 2024-04-27T03:52:17.554 INFO:teuthology.orchestra.run.smithi046.stdout:(4/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 97 kB/s | 31 kB 00:00 2024-04-27T03:52:17.688 INFO:teuthology.orchestra.run.smithi046.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 443 kB/s | 188 kB 00:00 2024-04-27T03:52:17.691 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:17.692 INFO:teuthology.orchestra.run.smithi046.stdout:Total 3.7 MB/s | 3.8 MB 00:01 2024-04-27T03:52:17.692 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-27T03:52:17.699 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-27T03:52:17.699 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-27T03:52:17.764 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-27T03:52:17.765 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-27T03:52:18.009 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-27T03:52:18.087 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-04-27T03:52:18.117 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-04-27T03:52:18.146 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1260.g5403a021.el9.n 3/5 2024-04-27T03:52:19.256 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1260.g5403a021.el9. 4/5 2024-04-27T03:52:19.291 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1260.g5403a021.el9.noarc 5/5 2024-04-27T03:52:19.940 INFO:teuthology.orchestra.run.smithi073.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 677 kB/s | 3.0 MB 00:04 2024-04-27T03:52:20.424 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1260.g5403a021.el9.noarc 5/5 2024-04-27T03:52:20.425 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1260.g5403a021.el9. 1/5 2024-04-27T03:52:20.425 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1260.g5403a021.el9.noarc 2/5 2024-04-27T03:52:20.425 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1260.g5403a021.el9.n 3/5 2024-04-27T03:52:20.425 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-04-27T03:52:20.655 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-04-27T03:52:20.656 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:20.656 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-27T03:52:20.656 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards-2:18.2.2-1260.g5403a021.el9.noarch 2024-04-27T03:52:20.656 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard-2:18.2.2-1260.g5403a021.el9.noarch 2024-04-27T03:52:20.656 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts-2:18.2.2-1260.g5403a021.el9.noarch 2024-04-27T03:52:20.656 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-27T03:52:20.656 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-27T03:52:20.656 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:20.656 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:20.849 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-27T03:52:20.874 INFO:teuthology.orchestra.run.smithi073.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 269 kB/s | 250 kB 00:00 2024-04-27T03:52:21.378 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:04 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:21.505 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:21.506 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-04-27T03:52:21.506 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-04-27T03:52:21.506 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-04-27T03:52:21.507 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-27T03:52:21.507 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1260.g5403a021.el9 ceph-noarch 7.4 M 2024-04-27T03:52:21.507 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-04-27T03:52:21.507 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-27T03:52:21.507 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-27T03:52:21.507 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-27T03:52:21.507 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-27T03:52:21.507 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-27T03:52:21.507 INFO:teuthology.orchestra.run.smithi046.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-27T03:52:21.507 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-27T03:52:21.507 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-27T03:52:21.507 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-27T03:52:21.507 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-27T03:52:21.508 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-27T03:52:21.508 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:21.508 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:52:21.508 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-04-27T03:52:21.508 INFO:teuthology.orchestra.run.smithi046.stdout:Install 12 Packages 2024-04-27T03:52:21.508 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:21.508 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 38 M 2024-04-27T03:52:21.509 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 209 M 2024-04-27T03:52:21.509 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-27T03:52:22.470 INFO:teuthology.orchestra.run.smithi046.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.2-12 12 MB/s | 7.4 MB 00:00 2024-04-27T03:52:22.528 INFO:teuthology.orchestra.run.smithi046.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 564 kB/s | 33 kB 00:00 2024-04-27T03:52:22.687 INFO:teuthology.orchestra.run.smithi046.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 19 MB/s | 3.0 MB 00:00 2024-04-27T03:52:22.737 INFO:teuthology.orchestra.run.smithi046.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 366 kB/s | 18 kB 00:00 2024-04-27T03:52:22.787 INFO:teuthology.orchestra.run.smithi046.stdout:(5/12): libquadmath-11.4.1-3.el9.x86_64.rpm 209 kB/s | 195 kB 00:00 2024-04-27T03:52:22.812 INFO:teuthology.orchestra.run.smithi046.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 468 kB/s | 35 kB 00:00 2024-04-27T03:52:22.840 INFO:teuthology.orchestra.run.smithi073.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 750 kB/s | 4.7 MB 00:06 2024-04-27T03:52:22.896 INFO:teuthology.orchestra.run.smithi046.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-04-27T03:52:23.029 INFO:teuthology.orchestra.run.smithi046.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-04-27T03:52:23.105 INFO:teuthology.orchestra.run.smithi046.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.3 MB/s | 250 kB 00:00 2024-04-27T03:52:23.172 INFO:teuthology.orchestra.run.smithi046.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 5.1 MB 00:00 2024-04-27T03:52:23.639 INFO:teuthology.orchestra.run.smithi046.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-04-27T03:52:23.672 INFO:teuthology.orchestra.run.smithi046.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 441 kB/s | 803 kB 00:01 2024-04-27T03:52:23.673 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:23.673 INFO:teuthology.orchestra.run.smithi046.stdout:Total 18 MB/s | 38 MB 00:02 2024-04-27T03:52:24.032 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-27T03:52:24.060 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-27T03:52:24.060 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-27T03:52:24.482 INFO:teuthology.orchestra.run.smithi073.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 750 kB/s | 5.1 MB 00:06 2024-04-27T03:52:24.492 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-27T03:52:24.493 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-27T03:52:25.274 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-27T03:52:25.335 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-27T03:52:25.379 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-27T03:52:25.491 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-27T03:52:25.530 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-27T03:52:25.714 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-27T03:52:25.746 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-27T03:52:25.824 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-27T03:52:26.507 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-27T03:52:26.613 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-27T03:52:27.919 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-27T03:52:29.000 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-27T03:52:29.027 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1260.g5403a 12/12 2024-04-27T03:52:31.692 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1260.g5403a 12/12 2024-04-27T03:52:31.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1260.g5403a 1/12 2024-04-27T03:52:31.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-27T03:52:31.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-27T03:52:31.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-27T03:52:31.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-27T03:52:31.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-27T03:52:31.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-27T03:52:31.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-27T03:52:31.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-27T03:52:31.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-27T03:52:31.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-27T03:52:32.110 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-27T03:52:32.110 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:32.111 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-27T03:52:32.111 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1260.g5403a021.el9.noarch 2024-04-27T03:52:32.111 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-27T03:52:32.111 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-27T03:52:32.111 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-27T03:52:32.111 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-27T03:52:32.111 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-27T03:52:32.111 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-27T03:52:32.111 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-27T03:52:32.111 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-27T03:52:32.111 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-27T03:52:32.111 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-27T03:52:32.111 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-27T03:52:32.112 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:32.112 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:32.481 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-rook 2024-04-27T03:52:33.014 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:16 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:33.139 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:33.141 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:33.141 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-04-27T03:52:33.141 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:33.141 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-27T03:52:33.141 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook noarch 2:18.2.2-1260.g5403a021.el9 ceph-noarch 49 k 2024-04-27T03:52:33.141 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-04-27T03:52:33.141 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-27T03:52:33.141 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-27T03:52:33.141 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-27T03:52:33.142 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-27T03:52:33.142 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-27T03:52:33.142 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-27T03:52:33.142 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-27T03:52:33.142 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-27T03:52:33.142 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-27T03:52:33.142 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-27T03:52:33.142 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-27T03:52:33.142 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-27T03:52:33.142 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:33.142 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:52:33.142 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:33.142 INFO:teuthology.orchestra.run.smithi046.stdout:Install 13 Packages 2024-04-27T03:52:33.143 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:33.143 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 2.2 M 2024-04-27T03:52:33.143 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 26 M 2024-04-27T03:52:33.143 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-27T03:52:33.410 INFO:teuthology.orchestra.run.smithi046.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 364 kB/s | 19 kB 00:00 2024-04-27T03:52:33.460 INFO:teuthology.orchestra.run.smithi046.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 249 kB/s | 26 kB 00:00 2024-04-27T03:52:33.527 INFO:teuthology.orchestra.run.smithi046.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-04-27T03:52:33.552 INFO:teuthology.orchestra.run.smithi046.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-04-27T03:52:33.627 INFO:teuthology.orchestra.run.smithi046.stdout:(5/13): python3-requests-oauthlib-1.3.0-12.el9. 716 kB/s | 54 kB 00:00 2024-04-27T03:52:33.653 INFO:teuthology.orchestra.run.smithi046.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.2 MB/s | 279 kB 00:00 2024-04-27T03:52:33.678 INFO:teuthology.orchestra.run.smithi046.stdout:(7/13): ceph-mgr-rook-18.2.2-1260.g5403a021.el9 154 kB/s | 49 kB 00:00 2024-04-27T03:52:33.720 INFO:teuthology.orchestra.run.smithi046.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 224 kB/s | 15 kB 00:00 2024-04-27T03:52:33.745 INFO:teuthology.orchestra.run.smithi046.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 274 kB/s | 32 kB 00:00 2024-04-27T03:52:33.812 INFO:teuthology.orchestra.run.smithi046.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 1.5 MB/s | 206 kB 00:00 2024-04-27T03:52:33.837 INFO:teuthology.orchestra.run.smithi046.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-04-27T03:52:33.862 INFO:teuthology.orchestra.run.smithi046.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-04-27T03:52:33.938 INFO:teuthology.orchestra.run.smithi046.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.7 MB/s | 1.0 MB 00:00 2024-04-27T03:52:33.939 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:33.939 INFO:teuthology.orchestra.run.smithi046.stdout:Total 2.8 MB/s | 2.2 MB 00:00 2024-04-27T03:52:34.004 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-27T03:52:34.020 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-27T03:52:34.020 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-27T03:52:34.111 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-27T03:52:34.111 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-27T03:52:34.692 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-27T03:52:34.876 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-27T03:52:34.941 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-27T03:52:34.997 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-27T03:52:35.033 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-27T03:52:35.075 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-27T03:52:35.172 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-27T03:52:35.291 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-27T03:52:35.343 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-27T03:52:36.100 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-27T03:52:36.155 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-27T03:52:36.190 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-27T03:52:36.239 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-27T03:52:36.267 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-rook-2:18.2.2-1260.g5403a021.el9.noarch 13/13 2024-04-27T03:52:37.023 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1260.g5403a021.el9.noarch 13/13 2024-04-27T03:52:37.024 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1260.g5403a021.el9.noarch 1/13 2024-04-27T03:52:37.024 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-27T03:52:37.024 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-27T03:52:37.024 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-27T03:52:37.024 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-27T03:52:37.024 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-27T03:52:37.024 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-27T03:52:37.024 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-27T03:52:37.024 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-27T03:52:37.024 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-27T03:52:37.024 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-27T03:52:37.024 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-27T03:52:37.329 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-27T03:52:37.329 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:37.329 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-27T03:52:37.329 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook-2:18.2.2-1260.g5403a021.el9.noarch 2024-04-27T03:52:37.329 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-27T03:52:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-27T03:52:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-27T03:52:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-27T03:52:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-27T03:52:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-27T03:52:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-27T03:52:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-27T03:52:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-27T03:52:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-27T03:52:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-27T03:52:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-27T03:52:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:37.330 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:37.568 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-cephadm 2024-04-27T03:52:38.112 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:21 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:38.236 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:38.237 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:38.237 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-04-27T03:52:38.237 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:38.237 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-27T03:52:38.237 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1260.g5403a021.el9 ceph-noarch 138 k 2024-04-27T03:52:38.237 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-04-27T03:52:38.237 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-27T03:52:38.237 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-27T03:52:38.238 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-27T03:52:38.238 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-27T03:52:38.238 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-27T03:52:38.238 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:38.238 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:52:38.238 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:38.238 INFO:teuthology.orchestra.run.smithi046.stdout:Install 6 Packages 2024-04-27T03:52:38.238 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:38.239 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 7.0 M 2024-04-27T03:52:38.239 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 33 M 2024-04-27T03:52:38.239 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-27T03:52:38.797 INFO:teuthology.orchestra.run.smithi046.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 981 kB/s | 248 kB 00:00 2024-04-27T03:52:38.906 INFO:teuthology.orchestra.run.smithi046.stdout:(2/6): ceph-mgr-cephadm-18.2.2-1260.g5403a021.e 382 kB/s | 138 kB 00:00 2024-04-27T03:52:39.256 INFO:teuthology.orchestra.run.smithi046.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.4 MB/s | 6.0 MB 00:00 2024-04-27T03:52:39.398 INFO:teuthology.orchestra.run.smithi046.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 124 kB/s | 58 kB 00:00 2024-04-27T03:52:39.624 INFO:teuthology.orchestra.run.smithi046.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 664 kB/s | 548 kB 00:00 2024-04-27T03:52:39.724 INFO:teuthology.orchestra.run.smithi046.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 109 kB/s | 51 kB 00:00 2024-04-27T03:52:39.727 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:39.727 INFO:teuthology.orchestra.run.smithi046.stdout:Total 4.7 MB/s | 7.0 MB 00:01 2024-04-27T03:52:39.801 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-27T03:52:39.814 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-27T03:52:39.814 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-27T03:52:39.915 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-27T03:52:39.919 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-27T03:52:40.238 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-27T03:52:40.407 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-27T03:52:40.460 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-27T03:52:41.013 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-27T03:52:41.097 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-27T03:52:41.176 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-27T03:52:41.204 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1260.g5403a021.el9.noarch 6/6 2024-04-27T03:52:41.655 INFO:teuthology.orchestra.run.smithi073.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 801 kB/s | 16 MB 00:20 2024-04-27T03:52:41.656 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:41.657 INFO:teuthology.orchestra.run.smithi073.stdout:Total 1.4 MB/s | 38 MB 00:27 2024-04-27T03:52:41.908 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1260.g5403a021.el9.noarch 6/6 2024-04-27T03:52:41.908 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1260.g5403a021.el9.noarch 1/6 2024-04-27T03:52:41.909 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-27T03:52:41.909 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-27T03:52:41.909 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-27T03:52:41.909 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-27T03:52:42.016 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-04-27T03:52:42.043 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-04-27T03:52:42.043 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-04-27T03:52:42.148 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-27T03:52:42.148 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:42.148 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-27T03:52:42.148 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm-2:18.2.2-1260.g5403a021.el9.noarch 2024-04-27T03:52:42.148 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-27T03:52:42.148 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-27T03:52:42.148 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-27T03:52:42.148 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-27T03:52:42.149 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-27T03:52:42.149 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:42.149 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:42.357 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-fuse 2024-04-27T03:52:42.446 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-04-27T03:52:42.446 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-04-27T03:52:42.894 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:25 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:43.018 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:43.019 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:43.019 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-04-27T03:52:43.019 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:43.019 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-27T03:52:43.019 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse x86_64 2:18.2.2-1260.g5403a021.el9 ceph 848 k 2024-04-27T03:52:43.019 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-04-27T03:52:43.019 INFO:teuthology.orchestra.run.smithi046.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-27T03:52:43.019 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:43.019 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:52:43.020 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:43.020 INFO:teuthology.orchestra.run.smithi046.stdout:Install 2 Packages 2024-04-27T03:52:43.020 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:43.020 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 929 k 2024-04-27T03:52:43.020 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 2.7 M 2024-04-27T03:52:43.020 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-27T03:52:43.196 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-04-27T03:52:43.251 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-27T03:52:43.267 INFO:teuthology.orchestra.run.smithi046.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 663 kB/s | 80 kB 00:00 2024-04-27T03:52:43.294 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-27T03:52:43.404 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-27T03:52:43.445 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-27T03:52:43.583 INFO:teuthology.orchestra.run.smithi046.stdout:(2/2): ceph-fuse-18.2.2-1260.g5403a021.el9.x86_ 1.9 MB/s | 848 kB 00:00 2024-04-27T03:52:43.584 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:43.584 INFO:teuthology.orchestra.run.smithi046.stdout:Total 1.6 MB/s | 929 kB 00:00 2024-04-27T03:52:43.596 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-27T03:52:43.619 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-27T03:52:43.619 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-27T03:52:43.625 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-27T03:52:43.653 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-27T03:52:43.700 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-27T03:52:43.701 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-27T03:52:43.729 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-27T03:52:43.886 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-27T03:52:44.011 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-27T03:52:44.083 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-fuse-2:18.2.2-1260.g5403a021.el9.x86_64 2/2 2024-04-27T03:52:44.398 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-27T03:52:44.509 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-27T03:52:44.741 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1260.g5403a021.el9.x86_64 2/2 2024-04-27T03:52:44.741 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-fuse-2:18.2.2-1260.g5403a021.el9.x86_64 1/2 2024-04-27T03:52:44.987 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-27T03:52:44.987 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:44.987 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-27T03:52:44.987 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-2:18.2.2-1260.g5403a021.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-27T03:52:44.987 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:44.987 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:45.223 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librados-devel 2024-04-27T03:52:45.747 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-27T03:52:45.752 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:28 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:45.878 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:45.879 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:45.879 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-04-27T03:52:45.879 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:45.879 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-27T03:52:45.879 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel x86_64 2:18.2.2-1260.g5403a021.el9 ceph 127 k 2024-04-27T03:52:45.879 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:45.879 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:52:45.879 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:45.879 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-04-27T03:52:45.879 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:45.880 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 127 k 2024-04-27T03:52:45.880 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 456 k 2024-04-27T03:52:45.880 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-27T03:52:46.107 INFO:teuthology.orchestra.run.smithi046.stdout:librados-devel-18.2.2-1260.g5403a021.el9.x86_64 558 kB/s | 127 kB 00:00 2024-04-27T03:52:46.107 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:46.107 INFO:teuthology.orchestra.run.smithi046.stdout:Total 554 kB/s | 127 kB 00:00 2024-04-27T03:52:46.107 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-27T03:52:46.113 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-27T03:52:46.114 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-27T03:52:46.181 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-27T03:52:46.182 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-27T03:52:46.335 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-27T03:52:46.446 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados-devel-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:52:46.769 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-27T03:52:46.802 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1260.g5403a 12/12 2024-04-27T03:52:46.834 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados-devel-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:52:47.084 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados-devel-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:52:47.085 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:47.085 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-27T03:52:47.085 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:52:47.085 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:47.085 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:47.316 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install libcephfs2 2024-04-27T03:52:47.859 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:30 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:47.941 INFO:teuthology.orchestra.run.smithi046.stdout:Package libcephfs2-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:52:47.985 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:47.986 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-04-27T03:52:47.986 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:48.054 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install libcephfs-devel 2024-04-27T03:52:48.584 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:31 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:48.707 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:48.708 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:48.708 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-04-27T03:52:48.708 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:48.708 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-27T03:52:48.708 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel x86_64 2:18.2.2-1260.g5403a021.el9 ceph 31 k 2024-04-27T03:52:48.709 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:48.709 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:52:48.709 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:48.709 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-04-27T03:52:48.709 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:48.709 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 31 k 2024-04-27T03:52:48.709 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 139 k 2024-04-27T03:52:48.709 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-27T03:52:48.943 INFO:teuthology.orchestra.run.smithi046.stdout:libcephfs-devel-18.2.2-1260.g5403a021.el9.x86_6 133 kB/s | 31 kB 00:00 2024-04-27T03:52:48.944 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:48.944 INFO:teuthology.orchestra.run.smithi046.stdout:Total 132 kB/s | 31 kB 00:00 2024-04-27T03:52:48.944 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-27T03:52:48.948 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-27T03:52:48.948 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-27T03:52:48.950 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-27T03:52:48.950 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-27T03:52:49.039 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-27T03:52:49.157 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs-devel-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:52:49.274 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1260.g5403a 12/12 2024-04-27T03:52:49.275 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1260.g5403a 1/12 2024-04-27T03:52:49.275 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-27T03:52:49.275 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-27T03:52:49.275 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-27T03:52:49.275 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-27T03:52:49.275 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-27T03:52:49.275 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-27T03:52:49.275 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-27T03:52:49.275 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-27T03:52:49.275 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-27T03:52:49.276 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-27T03:52:49.495 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:52:49.655 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-27T03:52:49.655 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:49.655 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-04-27T03:52:49.655 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1260.g5403a021.el9.noarch 2024-04-27T03:52:49.655 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-27T03:52:49.655 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-27T03:52:49.655 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-27T03:52:49.655 INFO:teuthology.orchestra.run.smithi073.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-27T03:52:49.655 INFO:teuthology.orchestra.run.smithi073.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-27T03:52:49.656 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-27T03:52:49.656 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-27T03:52:49.656 INFO:teuthology.orchestra.run.smithi073.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-27T03:52:49.656 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-27T03:52:49.656 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-27T03:52:49.656 INFO:teuthology.orchestra.run.smithi073.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-27T03:52:49.656 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:49.657 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:52:49.705 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs-devel-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:52:49.705 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-27T03:52:49.705 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:52:49.705 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:49.878 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librados2 2024-04-27T03:52:50.021 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-rook 2024-04-27T03:52:50.412 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:33 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:50.494 INFO:teuthology.orchestra.run.smithi046.stdout:Package librados2-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:52:50.536 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:37 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:52:50.537 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:50.538 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-04-27T03:52:50.538 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:50.603 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librbd1 2024-04-27T03:52:50.656 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:52:50.657 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:52:50.657 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-04-27T03:52:50.657 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:52:50.657 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-04-27T03:52:50.657 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-rook noarch 2:18.2.2-1260.g5403a021.el9 ceph-noarch 49 k 2024-04-27T03:52:50.657 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-04-27T03:52:50.657 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-27T03:52:50.658 INFO:teuthology.orchestra.run.smithi073.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-27T03:52:50.658 INFO:teuthology.orchestra.run.smithi073.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-27T03:52:50.658 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-27T03:52:50.658 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-27T03:52:50.658 INFO:teuthology.orchestra.run.smithi073.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-27T03:52:50.658 INFO:teuthology.orchestra.run.smithi073.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-27T03:52:50.658 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-27T03:52:50.658 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-27T03:52:50.658 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-27T03:52:50.658 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-27T03:52:50.658 INFO:teuthology.orchestra.run.smithi073.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-27T03:52:50.658 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:50.658 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:52:50.659 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:52:50.659 INFO:teuthology.orchestra.run.smithi073.stdout:Install 13 Packages 2024-04-27T03:52:50.659 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:50.659 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 2.2 M 2024-04-27T03:52:50.659 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 26 M 2024-04-27T03:52:50.659 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-04-27T03:52:51.135 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:34 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:51.216 INFO:teuthology.orchestra.run.smithi046.stdout:Package librbd1-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:52:51.259 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:51.260 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-04-27T03:52:51.260 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:51.331 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rados 2024-04-27T03:52:51.360 INFO:teuthology.orchestra.run.smithi073.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 209 kB/s | 19 kB 00:00 2024-04-27T03:52:51.385 INFO:teuthology.orchestra.run.smithi073.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 220 kB/s | 26 kB 00:00 2024-04-27T03:52:51.453 INFO:teuthology.orchestra.run.smithi073.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-04-27T03:52:51.494 INFO:teuthology.orchestra.run.smithi073.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-04-27T03:52:51.520 INFO:teuthology.orchestra.run.smithi073.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 279 kB 00:00 2024-04-27T03:52:51.545 INFO:teuthology.orchestra.run.smithi073.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-04-27T03:52:51.570 INFO:teuthology.orchestra.run.smithi073.stdout:(7/13): ceph-mgr-rook-18.2.2-1260.g5403a021.el9 163 kB/s | 49 kB 00:00 2024-04-27T03:52:51.865 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:34 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:51.895 INFO:teuthology.orchestra.run.smithi073.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 43 kB/s | 15 kB 00:00 2024-04-27T03:52:51.946 INFO:teuthology.orchestra.run.smithi073.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 76 kB/s | 32 kB 00:00 2024-04-27T03:52:51.948 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-rados-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:52:51.990 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:51.992 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-04-27T03:52:51.992 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:52.061 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rgw 2024-04-27T03:52:52.121 INFO:teuthology.orchestra.run.smithi073.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 338 kB/s | 59 kB 00:00 2024-04-27T03:52:52.155 INFO:teuthology.orchestra.run.smithi073.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 352 kB/s | 206 kB 00:00 2024-04-27T03:52:52.263 INFO:teuthology.orchestra.run.smithi073.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 632 kB/s | 90 kB 00:00 2024-04-27T03:52:52.405 INFO:teuthology.orchestra.run.smithi073.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2024-04-27T03:52:52.408 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:52.408 INFO:teuthology.orchestra.run.smithi073.stdout:Total 1.3 MB/s | 2.2 MB 00:01 2024-04-27T03:52:52.469 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-04-27T03:52:52.484 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-04-27T03:52:52.484 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-04-27T03:52:52.566 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-04-27T03:52:52.567 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-04-27T03:52:52.597 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:35 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:52.679 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-rgw-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:52:52.722 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:52.723 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-04-27T03:52:52.723 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:52.804 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-cephfs 2024-04-27T03:52:53.157 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-04-27T03:52:53.334 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:36 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:53.336 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-27T03:52:53.398 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-27T03:52:53.414 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-cephfs-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:52:53.453 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-27T03:52:53.457 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:53.459 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-04-27T03:52:53.459 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:53.482 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-27T03:52:53.523 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-27T03:52:53.531 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rbd 2024-04-27T03:52:53.617 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-27T03:52:53.727 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-27T03:52:53.776 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-27T03:52:54.070 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:37 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:54.152 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-rbd-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:52:54.194 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:54.196 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-04-27T03:52:54.196 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:54.271 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-fuse 2024-04-27T03:52:54.485 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-27T03:52:54.539 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-27T03:52:54.573 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-27T03:52:54.621 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-27T03:52:54.643 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-rook-2:18.2.2-1260.g5403a021.el9.noarch 13/13 2024-04-27T03:52:54.805 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:37 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:54.929 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:54.930 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:54.930 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-04-27T03:52:54.930 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:54.930 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-27T03:52:54.931 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse x86_64 2:18.2.2-1260.g5403a021.el9 ceph 86 k 2024-04-27T03:52:54.931 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:54.931 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:52:54.931 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:54.931 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-04-27T03:52:54.931 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:54.931 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 86 k 2024-04-27T03:52:54.931 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 227 k 2024-04-27T03:52:54.931 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-27T03:52:55.129 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-fuse-18.2.2-1260.g5403a021.el9.x86_64.rpm 437 kB/s | 86 kB 00:00 2024-04-27T03:52:55.130 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:55.131 INFO:teuthology.orchestra.run.smithi046.stdout:Total 434 kB/s | 86 kB 00:00 2024-04-27T03:52:55.131 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-27T03:52:55.138 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-27T03:52:55.138 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-27T03:52:55.204 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-27T03:52:55.204 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-27T03:52:55.350 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1260.g5403a021.el9.noarch 13/13 2024-04-27T03:52:55.350 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1260.g5403a021.el9.noarch 1/13 2024-04-27T03:52:55.350 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-27T03:52:55.350 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-27T03:52:55.351 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-27T03:52:55.351 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-27T03:52:55.351 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-27T03:52:55.351 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-27T03:52:55.351 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-27T03:52:55.351 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-27T03:52:55.352 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-27T03:52:55.352 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-27T03:52:55.352 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-27T03:52:55.354 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-27T03:52:55.452 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-fuse-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:52:55.686 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-27T03:52:55.686 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:55.686 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-04-27T03:52:55.686 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-rook-2:18.2.2-1260.g5403a021.el9.noarch 2024-04-27T03:52:55.686 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-27T03:52:55.687 INFO:teuthology.orchestra.run.smithi073.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-27T03:52:55.687 INFO:teuthology.orchestra.run.smithi073.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-27T03:52:55.687 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-27T03:52:55.687 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-27T03:52:55.687 INFO:teuthology.orchestra.run.smithi073.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-27T03:52:55.687 INFO:teuthology.orchestra.run.smithi073.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-27T03:52:55.687 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-27T03:52:55.687 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-27T03:52:55.687 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-27T03:52:55.687 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-27T03:52:55.687 INFO:teuthology.orchestra.run.smithi073.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-27T03:52:55.687 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:55.687 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:52:55.790 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:52:55.906 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-cephadm 2024-04-27T03:52:56.045 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-fuse-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:52:56.045 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:56.046 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-27T03:52:56.046 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:52:56.046 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:56.046 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:56.277 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-mirror 2024-04-27T03:52:56.416 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:43 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:52:56.536 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:52:56.537 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:52:56.537 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-04-27T03:52:56.538 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:52:56.538 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-04-27T03:52:56.538 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1260.g5403a021.el9 ceph-noarch 138 k 2024-04-27T03:52:56.538 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-04-27T03:52:56.538 INFO:teuthology.orchestra.run.smithi073.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-27T03:52:56.538 INFO:teuthology.orchestra.run.smithi073.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-27T03:52:56.539 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-27T03:52:56.539 INFO:teuthology.orchestra.run.smithi073.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-27T03:52:56.539 INFO:teuthology.orchestra.run.smithi073.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-27T03:52:56.539 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:56.539 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:52:56.539 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:52:56.540 INFO:teuthology.orchestra.run.smithi073.stdout:Install 6 Packages 2024-04-27T03:52:56.540 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:52:56.541 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 7.0 M 2024-04-27T03:52:56.541 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 33 M 2024-04-27T03:52:56.541 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-04-27T03:52:56.818 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:39 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:56.942 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:56.943 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:56.943 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-04-27T03:52:56.943 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:56.943 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-27T03:52:56.943 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror x86_64 2:18.2.2-1260.g5403a021.el9 ceph 3.0 M 2024-04-27T03:52:56.943 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:56.943 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:52:56.944 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:56.944 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-04-27T03:52:56.944 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:56.944 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 3.0 M 2024-04-27T03:52:56.944 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 12 M 2024-04-27T03:52:56.944 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-27T03:52:57.307 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-mirror-18.2.2-1260.g5403a021.el9.x86_64.rpm 8.2 MB/s | 3.0 MB 00:00 2024-04-27T03:52:57.308 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:57.308 INFO:teuthology.orchestra.run.smithi046.stdout:Total 8.2 MB/s | 3.0 MB 00:00 2024-04-27T03:52:57.308 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-27T03:52:57.319 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-27T03:52:57.319 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-27T03:52:57.382 INFO:teuthology.orchestra.run.smithi073.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-04-27T03:52:57.402 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-27T03:52:57.403 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-27T03:52:57.424 INFO:teuthology.orchestra.run.smithi073.stdout:(2/6): ceph-mgr-cephadm-18.2.2-1260.g5403a021.e 574 kB/s | 138 kB 00:00 2024-04-27T03:52:57.516 INFO:teuthology.orchestra.run.smithi073.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 630 kB/s | 58 kB 00:00 2024-04-27T03:52:57.541 INFO:teuthology.orchestra.run.smithi073.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.4 MB/s | 548 kB 00:00 2024-04-27T03:52:57.566 INFO:teuthology.orchestra.run.smithi073.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-04-27T03:52:57.750 INFO:teuthology.orchestra.run.smithi073.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-04-27T03:52:57.750 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:52:57.751 INFO:teuthology.orchestra.run.smithi073.stdout:Total 5.8 MB/s | 7.0 MB 00:01 2024-04-27T03:52:57.822 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-04-27T03:52:57.834 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-04-27T03:52:57.834 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-04-27T03:52:57.845 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-27T03:52:57.900 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-mirror-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:52:57.925 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:52:57.925 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T03:52:57.926 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-27T03:52:57.926 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-27T03:52:57.926 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-27T03:52:57.926 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:57.928 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-04-27T03:52:57.929 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-04-27T03:52:58.227 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-04-27T03:52:58.412 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-27T03:52:58.460 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-27T03:52:58.906 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-mirror-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:52:58.906 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:58.906 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-27T03:52:58.906 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:52:58.906 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:58.907 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:52:58.967 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-27T03:52:59.063 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-27T03:52:59.133 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-27T03:52:59.153 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-nbd 2024-04-27T03:52:59.156 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1260.g5403a021.el9.noarch 6/6 2024-04-27T03:52:59.691 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:42 ago on Sat 27 Apr 2024 03:51:17 AM UTC. 2024-04-27T03:52:59.815 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-27T03:52:59.816 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:59.816 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-04-27T03:52:59.816 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:59.816 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-27T03:52:59.816 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd x86_64 2:18.2.2-1260.g5403a021.el9 ceph 172 k 2024-04-27T03:52:59.817 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:59.817 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-27T03:52:59.817 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-27T03:52:59.817 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-04-27T03:52:59.817 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:52:59.817 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 172 k 2024-04-27T03:52:59.817 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 494 k 2024-04-27T03:52:59.817 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-27T03:52:59.858 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1260.g5403a021.el9.noarch 6/6 2024-04-27T03:52:59.858 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1260.g5403a021.el9.noarch 1/6 2024-04-27T03:52:59.858 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-27T03:52:59.858 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-27T03:52:59.858 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-27T03:52:59.858 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-27T03:53:00.044 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-nbd-18.2.2-1260.g5403a021.el9.x86_64.rpm 756 kB/s | 172 kB 00:00 2024-04-27T03:53:00.045 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:53:00.045 INFO:teuthology.orchestra.run.smithi046.stdout:Total 752 kB/s | 172 kB 00:00 2024-04-27T03:53:00.045 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-27T03:53:00.054 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-27T03:53:00.054 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-27T03:53:00.110 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-27T03:53:00.110 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:00.110 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-04-27T03:53:00.110 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-cephadm-2:18.2.2-1260.g5403a021.el9.noarch 2024-04-27T03:53:00.110 INFO:teuthology.orchestra.run.smithi073.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-27T03:53:00.111 INFO:teuthology.orchestra.run.smithi073.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-27T03:53:00.111 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-27T03:53:00.111 INFO:teuthology.orchestra.run.smithi073.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-27T03:53:00.111 INFO:teuthology.orchestra.run.smithi073.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-27T03:53:00.111 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:00.111 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:53:00.119 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-27T03:53:00.120 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-27T03:53:00.279 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-27T03:53:00.333 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-fuse 2024-04-27T03:53:00.384 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-nbd-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:00.787 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:00.842 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:47 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:53:00.961 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:53:00.961 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:00.961 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-04-27T03:53:00.961 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:00.961 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-04-27T03:53:00.961 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-fuse x86_64 2:18.2.2-1260.g5403a021.el9 ceph 848 k 2024-04-27T03:53:00.961 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-04-27T03:53:00.962 INFO:teuthology.orchestra.run.smithi073.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-27T03:53:00.962 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:00.962 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:53:00.962 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:00.962 INFO:teuthology.orchestra.run.smithi073.stdout:Install 2 Packages 2024-04-27T03:53:00.962 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:00.963 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 929 k 2024-04-27T03:53:00.963 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 2.7 M 2024-04-27T03:53:00.963 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-04-27T03:53:01.044 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-nbd-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:01.045 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:53:01.045 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-27T03:53:01.045 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:53:01.045 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:53:01.045 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-27T03:53:01.243 INFO:teuthology.orchestra.run.smithi073.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 511 kB/s | 80 kB 00:00 2024-04-27T03:53:01.290 DEBUG:teuthology.parallel:result is None 2024-04-27T03:53:01.527 INFO:teuthology.orchestra.run.smithi073.stdout:(2/2): ceph-fuse-18.2.2-1260.g5403a021.el9.x86_ 1.9 MB/s | 848 kB 00:00 2024-04-27T03:53:01.527 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:53:01.527 INFO:teuthology.orchestra.run.smithi073.stdout:Total 1.6 MB/s | 929 kB 00:00 2024-04-27T03:53:01.539 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-04-27T03:53:01.561 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-04-27T03:53:01.561 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-04-27T03:53:01.637 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-04-27T03:53:01.638 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-04-27T03:53:01.814 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-04-27T03:53:01.967 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-27T03:53:02.049 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-fuse-2:18.2.2-1260.g5403a021.el9.x86_64 2/2 2024-04-27T03:53:02.701 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1260.g5403a021.el9.x86_64 2/2 2024-04-27T03:53:02.702 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-fuse-2:18.2.2-1260.g5403a021.el9.x86_64 1/2 2024-04-27T03:53:02.943 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-27T03:53:02.943 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:02.943 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-04-27T03:53:02.943 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-fuse-2:18.2.2-1260.g5403a021.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-27T03:53:02.943 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:02.943 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:53:03.181 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install librados-devel 2024-04-27T03:53:03.696 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:50 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:53:03.820 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:53:03.821 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:03.821 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2024-04-27T03:53:03.821 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:03.821 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-04-27T03:53:03.821 INFO:teuthology.orchestra.run.smithi073.stdout: librados-devel x86_64 2:18.2.2-1260.g5403a021.el9 ceph 127 k 2024-04-27T03:53:03.821 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:03.821 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:53:03.821 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:03.821 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-04-27T03:53:03.821 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:03.821 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 127 k 2024-04-27T03:53:03.821 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 456 k 2024-04-27T03:53:03.821 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-04-27T03:53:04.047 INFO:teuthology.orchestra.run.smithi073.stdout:librados-devel-18.2.2-1260.g5403a021.el9.x86_64 562 kB/s | 127 kB 00:00 2024-04-27T03:53:04.048 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:53:04.048 INFO:teuthology.orchestra.run.smithi073.stdout:Total 557 kB/s | 127 kB 00:00 2024-04-27T03:53:04.048 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-04-27T03:53:04.054 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-04-27T03:53:04.054 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-04-27T03:53:04.118 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-04-27T03:53:04.121 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-04-27T03:53:04.265 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-04-27T03:53:04.360 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librados-devel-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:04.735 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados-devel-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:04.975 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados-devel-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:04.975 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:04.975 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-04-27T03:53:04.975 INFO:teuthology.orchestra.run.smithi073.stdout: librados-devel-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:53:04.976 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:04.976 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:53:05.240 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install libcephfs2 2024-04-27T03:53:05.762 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:52 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:53:05.840 INFO:teuthology.orchestra.run.smithi073.stdout:Package libcephfs2-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:53:05.882 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:53:05.883 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-04-27T03:53:05.883 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:53:05.957 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install libcephfs-devel 2024-04-27T03:53:06.467 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:53 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:53:06.587 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:53:06.587 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:06.587 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2024-04-27T03:53:06.588 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:06.588 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-04-27T03:53:06.588 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs-devel x86_64 2:18.2.2-1260.g5403a021.el9 ceph 31 k 2024-04-27T03:53:06.588 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:06.588 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:53:06.588 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:06.588 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-04-27T03:53:06.588 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:06.588 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 31 k 2024-04-27T03:53:06.589 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 139 k 2024-04-27T03:53:06.589 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-04-27T03:53:06.934 INFO:teuthology.orchestra.run.smithi073.stdout:libcephfs-devel-18.2.2-1260.g5403a021.el9.x86_6 90 kB/s | 31 kB 00:00 2024-04-27T03:53:06.935 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:53:06.935 INFO:teuthology.orchestra.run.smithi073.stdout:Total 89 kB/s | 31 kB 00:00 2024-04-27T03:53:06.935 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-04-27T03:53:06.938 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-04-27T03:53:06.939 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-04-27T03:53:06.940 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-04-27T03:53:06.941 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-04-27T03:53:07.029 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-04-27T03:53:07.154 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephfs-devel-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:07.498 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:07.725 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephfs-devel-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:07.725 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:07.725 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-04-27T03:53:07.725 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs-devel-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:53:07.725 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:07.725 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:53:07.879 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install librados2 2024-04-27T03:53:08.388 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:55 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:53:08.465 INFO:teuthology.orchestra.run.smithi073.stdout:Package librados2-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:53:08.505 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:53:08.506 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-04-27T03:53:08.507 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:53:08.568 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install librbd1 2024-04-27T03:53:09.081 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:56 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:53:09.159 INFO:teuthology.orchestra.run.smithi073.stdout:Package librbd1-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:53:09.200 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:53:09.201 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-04-27T03:53:09.201 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:53:09.260 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-rados 2024-04-27T03:53:09.766 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:56 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:53:09.844 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-rados-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:53:09.884 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:53:09.885 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-04-27T03:53:09.885 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:53:09.947 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-rgw 2024-04-27T03:53:10.457 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:57 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:53:10.537 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-rgw-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:53:10.576 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:53:10.578 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-04-27T03:53:10.578 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:53:10.647 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-cephfs 2024-04-27T03:53:11.154 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:58 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:53:11.233 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-cephfs-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:53:11.273 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:53:11.274 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-04-27T03:53:11.274 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:53:11.338 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-rbd 2024-04-27T03:53:11.846 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:58 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:53:11.925 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-rbd-2:18.2.2-1260.g5403a021.el9.x86_64 is already installed. 2024-04-27T03:53:11.966 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:53:11.967 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-04-27T03:53:11.967 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:53:12.039 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install rbd-fuse 2024-04-27T03:53:12.547 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:59 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:53:12.665 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:53:12.665 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:12.666 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2024-04-27T03:53:12.666 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:12.666 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-04-27T03:53:12.666 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-fuse x86_64 2:18.2.2-1260.g5403a021.el9 ceph 86 k 2024-04-27T03:53:12.666 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:12.666 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:53:12.666 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:12.666 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-04-27T03:53:12.666 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:12.667 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 86 k 2024-04-27T03:53:12.667 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 227 k 2024-04-27T03:53:12.667 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-04-27T03:53:12.979 INFO:teuthology.orchestra.run.smithi073.stdout:rbd-fuse-18.2.2-1260.g5403a021.el9.x86_64.rpm 278 kB/s | 86 kB 00:00 2024-04-27T03:53:12.979 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:53:12.980 INFO:teuthology.orchestra.run.smithi073.stdout:Total 276 kB/s | 86 kB 00:00 2024-04-27T03:53:12.980 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-04-27T03:53:12.987 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-04-27T03:53:12.987 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-04-27T03:53:13.049 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-04-27T03:53:13.049 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-04-27T03:53:13.198 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-04-27T03:53:13.290 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-fuse-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:13.625 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:13.874 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-fuse-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:13.874 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:13.874 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-04-27T03:53:13.874 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-fuse-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:53:13.874 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:13.874 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:53:14.079 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install rbd-mirror 2024-04-27T03:53:14.600 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:01 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:53:14.723 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:53:14.724 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:14.724 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2024-04-27T03:53:14.724 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:14.725 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-04-27T03:53:14.725 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-mirror x86_64 2:18.2.2-1260.g5403a021.el9 ceph 3.0 M 2024-04-27T03:53:14.725 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:14.725 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:53:14.725 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:14.725 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-04-27T03:53:14.725 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:14.725 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 3.0 M 2024-04-27T03:53:14.725 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 12 M 2024-04-27T03:53:14.725 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-04-27T03:53:18.167 INFO:teuthology.orchestra.run.smithi073.stdout:rbd-mirror-18.2.2-1260.g5403a021.el9.x86_64.rpm 889 kB/s | 3.0 MB 00:03 2024-04-27T03:53:18.168 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:53:18.168 INFO:teuthology.orchestra.run.smithi073.stdout:Total 889 kB/s | 3.0 MB 00:03 2024-04-27T03:53:18.168 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-04-27T03:53:18.178 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-04-27T03:53:18.178 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-04-27T03:53:18.261 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-04-27T03:53:18.261 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-04-27T03:53:18.682 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-04-27T03:53:18.743 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-mirror-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:18.768 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:18.768 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T03:53:18.768 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-27T03:53:18.768 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-27T03:53:18.769 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-27T03:53:18.769 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:19.750 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-mirror-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:19.750 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:19.750 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-04-27T03:53:19.750 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-mirror-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:53:19.750 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:19.750 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:53:19.987 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install rbd-nbd 2024-04-27T03:53:20.505 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:07 ago on Sat 27 Apr 2024 03:51:13 AM UTC. 2024-04-27T03:53:20.628 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-04-27T03:53:20.629 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:20.629 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2024-04-27T03:53:20.629 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:20.629 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-04-27T03:53:20.629 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-nbd x86_64 2:18.2.2-1260.g5403a021.el9 ceph 172 k 2024-04-27T03:53:20.629 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:20.630 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-04-27T03:53:20.630 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-04-27T03:53:20.630 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-04-27T03:53:20.630 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:20.630 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 172 k 2024-04-27T03:53:20.630 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 494 k 2024-04-27T03:53:20.630 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-04-27T03:53:20.880 INFO:teuthology.orchestra.run.smithi073.stdout:rbd-nbd-18.2.2-1260.g5403a021.el9.x86_64.rpm 688 kB/s | 172 kB 00:00 2024-04-27T03:53:20.881 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:53:20.881 INFO:teuthology.orchestra.run.smithi073.stdout:Total 683 kB/s | 172 kB 00:00 2024-04-27T03:53:20.881 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-04-27T03:53:20.889 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-04-27T03:53:20.889 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-04-27T03:53:20.951 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-04-27T03:53:20.952 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-04-27T03:53:21.108 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-04-27T03:53:21.201 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-nbd-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:21.605 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:21.858 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-nbd-2:18.2.2-1260.g5403a021.el9.x86_64 1/1 2024-04-27T03:53:21.858 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:21.858 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-04-27T03:53:21.858 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-nbd-2:18.2.2-1260.g5403a021.el9.x86_64 2024-04-27T03:53:21.858 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:53:21.858 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-04-27T03:53:22.101 DEBUG:teuthology.parallel:result is None 2024-04-27T03:53:22.102 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-27T03:53:22.102 INFO:teuthology.packaging:ref: None 2024-04-27T03:53:22.102 INFO:teuthology.packaging:tag: None 2024-04-27T03:53:22.102 INFO:teuthology.packaging:branch: reef 2024-04-27T03:53:22.102 INFO:teuthology.packaging:sha1: 4e213a58e787437ff508e6c890ccd5fd32dc96b5 2024-04-27T03:53:22.102 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-04-27T03:53:22.343 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-27T03:53:22.370 INFO:teuthology.orchestra.run.smithi046.stdout:18.2.2-1260.g5403a021.el9 2024-04-27T03:53:22.370 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1260.g5403a021.el9 2024-04-27T03:53:22.370 INFO:teuthology.task.install:The correct ceph version 18.2.2-1260.g5403a021 is installed. 2024-04-27T03:53:22.372 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-27T03:53:22.372 INFO:teuthology.packaging:ref: None 2024-04-27T03:53:22.372 INFO:teuthology.packaging:tag: None 2024-04-27T03:53:22.372 INFO:teuthology.packaging:branch: reef 2024-04-27T03:53:22.373 INFO:teuthology.packaging:sha1: 4e213a58e787437ff508e6c890ccd5fd32dc96b5 2024-04-27T03:53:22.373 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-04-27T03:53:22.680 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-27T03:53:22.701 INFO:teuthology.orchestra.run.smithi073.stdout:18.2.2-1260.g5403a021.el9 2024-04-27T03:53:22.701 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1260.g5403a021.el9 2024-04-27T03:53:22.701 INFO:teuthology.task.install:The correct ceph version 18.2.2-1260.g5403a021 is installed. 2024-04-27T03:53:22.703 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-27T03:53:22.703 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T03:53:22.703 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-27T03:53:22.734 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-04-27T03:53:22.734 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-27T03:53:22.766 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-27T03:53:22.766 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T03:53:22.766 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2024-04-27T03:53:22.803 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-27T03:53:22.871 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-04-27T03:53:22.871 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/daemon-helper 2024-04-27T03:53:22.902 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-27T03:53:22.966 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-27T03:53:22.966 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T03:53:22.966 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-27T03:53:22.994 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-27T03:53:23.059 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-04-27T03:53:23.059 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-27T03:53:23.083 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-27T03:53:23.146 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-27T03:53:23.146 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T03:53:23.146 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2024-04-27T03:53:23.171 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-27T03:53:23.236 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-04-27T03:53:23.236 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/stdin-killer 2024-04-27T03:53:23.259 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-27T03:53:23.323 INFO:teuthology.run_tasks:Running task print... 2024-04-27T03:53:23.332 INFO:teuthology.task.print:**** done install task... 2024-04-27T03:53:23.332 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-27T03:53:23.432 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '4e213a58e787437ff508e6c890ccd5fd32dc96b5'} 2024-04-27T03:53:23.433 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-04-27T03:53:23.433 INFO:tasks.cephadm:Cluster fsid is b1d3a2d0-0449-11ef-bc93-c7b262605968 2024-04-27T03:53:23.433 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-27T03:53:23.433 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-27T03:53:23.434 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi046': '172.21.15.46', 'mon.smithi073': '172.21.15.73'} 2024-04-27T03:53:23.434 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-27T03:53:23.434 DEBUG:teuthology.orchestra.run.smithi046:> sudo hostname $(hostname -s) 2024-04-27T03:53:23.459 DEBUG:teuthology.orchestra.run.smithi073:> sudo hostname $(hostname -s) 2024-04-27T03:53:23.484 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-04-27T03:53:23.484 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e213a58e787437ff508e6c890ccd5fd32dc96b5 2024-04-27T03:53:23.706 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '4e213a58e787437ff508e6c890ccd5fd32dc96b5', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54989/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3253-g4e213a58', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3253.g4e213a58'}, 'url': 'https://1.chacra.ceph.com/r/ceph/main/4e213a58e787437ff508e6c890ccd5fd32dc96b5/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 23:04:47.839086', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/main/4e213a58e787437ff508e6c890ccd5fd32dc96b5/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-27T03:53:23.941 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref reef, sha1 5403a02127f27ec6c3309ef057522ab70cdeb9d2 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-04-27T03:53:23.943 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/reef/5403a02127f27ec6c3309ef057522ab70cdeb9d2/centos/9/x86_64/flavors/default/cephadm 2024-04-27T03:53:23.944 DEBUG:teuthology.orchestra.run.smithi046:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/reef/5403a02127f27ec6c3309ef057522ab70cdeb9d2/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-27T03:53:24.554 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r--. 1 ubuntu ubuntu 215333 Apr 27 03:53 /home/ubuntu/cephtest/cephadm 2024-04-27T03:53:24.554 DEBUG:teuthology.orchestra.run.smithi073:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/reef/5403a02127f27ec6c3309ef057522ab70cdeb9d2/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-27T03:53:24.970 INFO:teuthology.orchestra.run.smithi073.stdout:-rw-r--r--. 1 ubuntu ubuntu 215333 Apr 27 03:53 /home/ubuntu/cephtest/cephadm 2024-04-27T03:53:24.970 DEBUG:teuthology.orchestra.run.smithi046:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-27T03:53:24.989 DEBUG:teuthology.orchestra.run.smithi073:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-27T03:53:25.021 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-04-27T03:53:25.021 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-04-27T03:53:25.032 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-04-27T03:53:26.487 INFO:teuthology.orchestra.run.smithi046.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-04-27T03:53:26.512 INFO:teuthology.orchestra.run.smithi073.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-04-27T03:53:57.368 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-04-27T03:53:57.369 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph_version": "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)", 2024-04-27T03:53:57.369 INFO:teuthology.orchestra.run.smithi073.stdout: "image_id": "267a473d505df5b518068223eb9431973cc4aab49462dd79c898418ebe6d341a", 2024-04-27T03:53:57.369 INFO:teuthology.orchestra.run.smithi073.stdout: "repo_digests": [ 2024-04-27T03:53:57.369 INFO:teuthology.orchestra.run.smithi073.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec" 2024-04-27T03:53:57.369 INFO:teuthology.orchestra.run.smithi073.stdout: ] 2024-04-27T03:53:57.369 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-04-27T03:53:59.440 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-27T03:53:59.440 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)", 2024-04-27T03:53:59.440 INFO:teuthology.orchestra.run.smithi046.stdout: "image_id": "267a473d505df5b518068223eb9431973cc4aab49462dd79c898418ebe6d341a", 2024-04-27T03:53:59.440 INFO:teuthology.orchestra.run.smithi046.stdout: "repo_digests": [ 2024-04-27T03:53:59.440 INFO:teuthology.orchestra.run.smithi046.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec" 2024-04-27T03:53:59.441 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-04-27T03:53:59.441 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-27T03:53:59.459 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph 2024-04-27T03:53:59.510 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /etc/ceph 2024-04-27T03:53:59.543 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /etc/ceph 2024-04-27T03:53:59.568 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 777 /etc/ceph 2024-04-27T03:53:59.610 INFO:tasks.cephadm:Writing seed config... 2024-04-27T03:53:59.611 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-27T03:53:59.611 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-27T03:53:59.611 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-27T03:53:59.611 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-27T03:53:59.611 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-27T03:53:59.611 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-27T03:53:59.611 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-27T03:53:59.612 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-27T03:53:59.612 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-27T03:53:59.612 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-27T03:53:59.612 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-27T03:53:59.612 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-27T03:53:59.612 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-27T03:53:59.612 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-27T03:53:59.612 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-27T03:53:59.612 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-27T03:53:59.612 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-27T03:53:59.612 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-27T03:53:59.612 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-27T03:53:59.612 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-27T03:53:59.612 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-27T03:53:59.613 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-27T03:53:59.613 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-27T03:53:59.613 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-27T03:53:59.613 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-27T03:53:59.613 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-27T03:53:59.613 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-27T03:53:59.613 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-27T03:53:59.613 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-27T03:53:59.613 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-27T03:53:59.613 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-27T03:53:59.614 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-27T03:53:59.614 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-27T03:53:59.614 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-27T03:53:59.614 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-27T03:53:59.614 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-27T03:53:59.614 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-27T03:53:59.614 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-27T03:53:59.614 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-27T03:53:59.614 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-27T03:53:59.614 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-27T03:53:59.615 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-27T03:53:59.615 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T03:53:59.615 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-27T03:53:59.631 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = b1d3a2d0-0449-11ef-bc93-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-27T03:53:59.631 DEBUG:teuthology.orchestra.run.smithi046:mon.smithi046> sudo journalctl -f -n 0 -u ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@mon.smithi046.service 2024-04-27T03:53:59.674 INFO:tasks.cephadm:Bootstrapping... 2024-04-27T03:53:59.674 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.46 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-27T03:53:59.833 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-27T03:53:59.833 INFO:teuthology.orchestra.run.smithi046.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', 'b1d3a2d0-0449-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.46', '--skip-admin-label'] 2024-04-27T03:53:59.858 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-27T03:53:59.858 INFO:teuthology.orchestra.run.smithi046.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-27T03:53:59.859 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying podman|docker is present... 2024-04-27T03:53:59.882 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-27T03:53:59.883 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying lvm2 is present... 2024-04-27T03:53:59.883 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying time synchronization is in place... 2024-04-27T03:53:59.890 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-27T03:53:59.891 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-27T03:53:59.896 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-27T03:53:59.896 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-04-27T03:53:59.903 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout enabled 2024-04-27T03:53:59.909 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout active 2024-04-27T03:53:59.909 INFO:teuthology.orchestra.run.smithi046.stdout:Unit chronyd.service is enabled and running 2024-04-27T03:53:59.909 INFO:teuthology.orchestra.run.smithi046.stdout:Repeating the final host check... 2024-04-27T03:53:59.942 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-27T03:53:59.942 INFO:teuthology.orchestra.run.smithi046.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-27T03:53:59.942 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl is present 2024-04-27T03:53:59.942 INFO:teuthology.orchestra.run.smithi046.stdout:lvcreate is present 2024-04-27T03:53:59.948 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-27T03:53:59.948 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-27T03:53:59.954 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-27T03:53:59.954 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-04-27T03:53:59.961 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout enabled 2024-04-27T03:53:59.967 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout active 2024-04-27T03:53:59.967 INFO:teuthology.orchestra.run.smithi046.stdout:Unit chronyd.service is enabled and running 2024-04-27T03:53:59.967 INFO:teuthology.orchestra.run.smithi046.stdout:Host looks OK 2024-04-27T03:53:59.967 INFO:teuthology.orchestra.run.smithi046.stdout:Cluster fsid: b1d3a2d0-0449-11ef-bc93-c7b262605968 2024-04-27T03:53:59.967 INFO:teuthology.orchestra.run.smithi046.stdout:Acquiring lock 140012714636288 on /run/cephadm/b1d3a2d0-0449-11ef-bc93-c7b262605968.lock 2024-04-27T03:53:59.967 INFO:teuthology.orchestra.run.smithi046.stdout:Lock 140012714636288 acquired on /run/cephadm/b1d3a2d0-0449-11ef-bc93-c7b262605968.lock 2024-04-27T03:53:59.968 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying IP 172.21.15.46 port 3300 ... 2024-04-27T03:53:59.968 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying IP 172.21.15.46 port 6789 ... 2024-04-27T03:53:59.968 INFO:teuthology.orchestra.run.smithi046.stdout:Base mon IP(s) is [172.21.15.46:3300, 172.21.15.46:6789], mon addrv is [v2:172.21.15.46:3300,v1:172.21.15.46:6789] 2024-04-27T03:53:59.971 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.46 metric 100 2024-04-27T03:53:59.972 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.46 metric 100 2024-04-27T03:53:59.974 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-27T03:53:59.974 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-27T03:53:59.974 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-27T03:53:59.977 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-27T03:53:59.977 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-27T03:53:59.977 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-27T03:53:59.977 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-27T03:53:59.977 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-04-27T03:53:59.977 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-27T03:53:59.978 INFO:teuthology.orchestra.run.smithi046.stdout:Mon IP `172.21.15.46` is in CIDR network `172.21.0.0/20` 2024-04-27T03:53:59.978 INFO:teuthology.orchestra.run.smithi046.stdout:Mon IP `172.21.15.46` is in CIDR network `172.21.0.0/20` 2024-04-27T03:53:59.978 INFO:teuthology.orchestra.run.smithi046.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-27T03:53:59.978 INFO:teuthology.orchestra.run.smithi046.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-27T03:53:59.979 INFO:teuthology.orchestra.run.smithi046.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-04-27T03:54:00.493 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout 267a473d505df5b518068223eb9431973cc4aab49462dd79c898418ebe6d341a 2024-04-27T03:54:00.493 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-04-27T03:54:00.493 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Getting image source signatures 2024-04-27T03:54:00.493 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Copying blob sha256:c32a87db12de0536b4b006c4956fa4160c5ded9cb5e1e221fd52732710683ab0 2024-04-27T03:54:00.493 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-27T03:54:00.493 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Copying config sha256:267a473d505df5b518068223eb9431973cc4aab49462dd79c898418ebe6d341a 2024-04-27T03:54:00.493 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-27T03:54:01.423 INFO:teuthology.orchestra.run.smithi046.stdout:ceph: stdout ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable) 2024-04-27T03:54:01.424 INFO:teuthology.orchestra.run.smithi046.stdout:Ceph version: ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable) 2024-04-27T03:54:01.424 INFO:teuthology.orchestra.run.smithi046.stdout:Extracting ceph user uid/gid from container image... 2024-04-27T03:54:02.349 INFO:teuthology.orchestra.run.smithi046.stdout:stat: stdout 167 167 2024-04-27T03:54:02.349 INFO:teuthology.orchestra.run.smithi046.stdout:Creating initial keys... 2024-04-27T03:54:03.230 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph-authtool: stdout AQDadixmGuqQKBAAorgvDhhJMIdGt/jLFnfWuw== 2024-04-27T03:54:04.156 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph-authtool: stdout AQDbdixm6k2VIRAA3XukMOkqkC9qP+ZPTKy4Iw== 2024-04-27T03:54:05.145 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph-authtool: stdout AQDcdixmpAwyHhAA0q9vdaJJZHFWCGQPLGVTFw== 2024-04-27T03:54:05.146 INFO:teuthology.orchestra.run.smithi046.stdout:Creating initial monmap... 2024-04-27T03:54:06.085 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-27T03:54:06.086 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-04-27T03:54:06.086 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to b1d3a2d0-0449-11ef-bc93-c7b262605968 2024-04-27T03:54:06.086 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-27T03:54:06.086 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool for smithi046 [v2:172.21.15.46:3300,v1:172.21.15.46:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-27T03:54:06.086 INFO:teuthology.orchestra.run.smithi046.stdout:setting min_mon_release = pacific 2024-04-27T03:54:06.086 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: set fsid to b1d3a2d0-0449-11ef-bc93-c7b262605968 2024-04-27T03:54:06.086 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-27T03:54:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:54:06.086 INFO:teuthology.orchestra.run.smithi046.stdout:Creating mon... 2024-04-27T03:54:06.993 INFO:teuthology.orchestra.run.smithi046.stdout:create mon.smithi046 on 2024-04-27T03:54:07.195 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-27T03:54:07.360 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-27T03:54:07.551 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968.target. 2024-04-27T03:54:07.551 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968.target. 2024-04-27T03:54:07.782 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@mon.smithi046 2024-04-27T03:54:07.782 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@mon.smithi046.service: Unit ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@mon.smithi046.service not loaded. 2024-04-27T03:54:07.960 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968.target.wants/ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@mon.smithi046.service → /etc/systemd/system/ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@.service. 2024-04-27T03:54:08.558 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-27T03:54:08.559 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2024-04-27T03:54:08.565 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-27T03:54:08.565 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-04-27T03:54:08.566 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2024-04-27T03:54:08.566 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to enable service . firewalld.service is not available 2024-04-27T03:54:08.566 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mon to start... 2024-04-27T03:54:08.566 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mon... 2024-04-27T03:54:08.595 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:08 smithi046 systemd[1]: Started Ceph mon.smithi046 for b1d3a2d0-0449-11ef-bc93-c7b262605968. 2024-04-27T03:54:09.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout cluster: 2024-04-27T03:54:09.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout id: b1d3a2d0-0449-11ef-bc93-c7b262605968 2024-04-27T03:54:09.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-27T03:54:09.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-27T03:54:09.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout services: 2024-04-27T03:54:09.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi046 (age 0.591472s) 2024-04-27T03:54:09.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-27T03:54:09.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-27T03:54:09.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-27T03:54:09.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout data: 2024-04-27T03:54:09.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-27T03:54:09.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-27T03:54:09.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-27T03:54:09.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout pgs: 2024-04-27T03:54:09.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-27T03:54:09.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.012+0000 7f248f8ba700 1 Processor -- start 2024-04-27T03:54:09.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.012+0000 7f248f8ba700 1 -- start start 2024-04-27T03:54:09.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.013+0000 7f248f8ba700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24881065f0 0x7f2488106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:09.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.013+0000 7f248f8ba700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2488106fe0 con 0x7f24881065f0 2024-04-27T03:54:09.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.013+0000 7f248d656700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24881065f0 0x7f2488106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:09.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.013+0000 7f248d656700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24881065f0 0x7f2488106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47032/0 (socket says 172.21.15.46:47032) 2024-04-27T03:54:09.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.013+0000 7f248d656700 1 -- 172.21.15.46:0/3100821473 learned_addr learned my addr 172.21.15.46:0/3100821473 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:09.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.013+0000 7f248d656700 1 -- 172.21.15.46:0/3100821473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24881077a0 con 0x7f24881065f0 2024-04-27T03:54:09.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.014+0000 7f248d656700 1 --2- 172.21.15.46:0/3100821473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24881065f0 0x7f2488106a10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f24780098a0 tx=0x7f2478009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=612d696ed415b15d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:09.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.014+0000 7f247ffff700 1 -- 172.21.15.46:0/3100821473 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2478004180 con 0x7f24881065f0 2024-04-27T03:54:09.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.014+0000 7f247ffff700 1 -- 172.21.15.46:0/3100821473 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24780042e0 con 0x7f24881065f0 2024-04-27T03:54:09.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.014+0000 7f247ffff700 1 -- 172.21.15.46:0/3100821473 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24780036a0 con 0x7f24881065f0 2024-04-27T03:54:09.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.014+0000 7f248f8ba700 1 -- 172.21.15.46:0/3100821473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24881065f0 msgr2=0x7f2488106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:09.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.014+0000 7f248f8ba700 1 --2- 172.21.15.46:0/3100821473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24881065f0 0x7f2488106a10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f24780098a0 tx=0x7f2478009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.014+0000 7f248f8ba700 1 -- 172.21.15.46:0/3100821473 shutdown_connections 2024-04-27T03:54:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.014+0000 7f248f8ba700 1 --2- 172.21.15.46:0/3100821473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24881065f0 0x7f2488106a10 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.014+0000 7f248f8ba700 1 -- 172.21.15.46:0/3100821473 >> 172.21.15.46:0/3100821473 conn(0x7f2488101b50 msgr2=0x7f2488103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.015+0000 7f248f8ba700 1 -- 172.21.15.46:0/3100821473 shutdown_connections 2024-04-27T03:54:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.015+0000 7f248f8ba700 1 -- 172.21.15.46:0/3100821473 wait complete. 2024-04-27T03:54:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.015+0000 7f248f8ba700 1 Processor -- start 2024-04-27T03:54:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.015+0000 7f248f8ba700 1 -- start start 2024-04-27T03:54:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.015+0000 7f248f8ba700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24881065f0 0x7f248819bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.015+0000 7f248f8ba700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248819c220 con 0x7f24881065f0 2024-04-27T03:54:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.016+0000 7f248d656700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24881065f0 0x7f248819bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.016+0000 7f248d656700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24881065f0 0x7f248819bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47036/0 (socket says 172.21.15.46:47036) 2024-04-27T03:54:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.016+0000 7f248d656700 1 -- 172.21.15.46:0/4098744338 learned_addr learned my addr 172.21.15.46:0/4098744338 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.016+0000 7f248d656700 1 -- 172.21.15.46:0/4098744338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2478009550 con 0x7f24881065f0 2024-04-27T03:54:09.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.016+0000 7f248d656700 1 --2- 172.21.15.46:0/4098744338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24881065f0 0x7f248819bce0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2478004000 tx=0x7f2478004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:09.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.016+0000 7f247e7fc700 1 -- 172.21.15.46:0/4098744338 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2478003ce0 con 0x7f24881065f0 2024-04-27T03:54:09.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.016+0000 7f247e7fc700 1 -- 172.21.15.46:0/4098744338 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247801f070 con 0x7f24881065f0 2024-04-27T03:54:09.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.016+0000 7f248f8ba700 1 -- 172.21.15.46:0/4098744338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248819c420 con 0x7f24881065f0 2024-04-27T03:54:09.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.016+0000 7f247e7fc700 1 -- 172.21.15.46:0/4098744338 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2478011630 con 0x7f24881065f0 2024-04-27T03:54:09.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.016+0000 7f248f8ba700 1 -- 172.21.15.46:0/4098744338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248819c8c0 con 0x7f24881065f0 2024-04-27T03:54:09.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.017+0000 7f247e7fc700 1 -- 172.21.15.46:0/4098744338 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f2478021030 con 0x7f24881065f0 2024-04-27T03:54:09.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.017+0000 7f247e7fc700 1 -- 172.21.15.46:0/4098744338 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2478024d00 con 0x7f24881065f0 2024-04-27T03:54:09.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.017+0000 7f248f8ba700 1 -- 172.21.15.46:0/4098744338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2488107b10 con 0x7f24881065f0 2024-04-27T03:54:09.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.020+0000 7f247e7fc700 1 -- 172.21.15.46:0/4098744338 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f2478024480 con 0x7f24881065f0 2024-04-27T03:54:09.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.092+0000 7f248f8ba700 1 -- 172.21.15.46:0/4098744338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f248819f150 con 0x7f24881065f0 2024-04-27T03:54:09.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.092+0000 7f247e7fc700 1 -- 172.21.15.46:0/4098744338 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f2478011c20 con 0x7f24881065f0 2024-04-27T03:54:09.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.093+0000 7f248f8ba700 1 -- 172.21.15.46:0/4098744338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24881065f0 msgr2=0x7f248819bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:09.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.093+0000 7f248f8ba700 1 --2- 172.21.15.46:0/4098744338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24881065f0 0x7f248819bce0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2478004000 tx=0x7f2478004750 comp rx=0 tx=0).stop 2024-04-27T03:54:09.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.093+0000 7f248f8ba700 1 -- 172.21.15.46:0/4098744338 shutdown_connections 2024-04-27T03:54:09.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.093+0000 7f248f8ba700 1 --2- 172.21.15.46:0/4098744338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24881065f0 0x7f248819bce0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:09.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.093+0000 7f248f8ba700 1 -- 172.21.15.46:0/4098744338 >> 172.21.15.46:0/4098744338 conn(0x7f2488101b50 msgr2=0x7f248807b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:09.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.093+0000 7f248f8ba700 1 -- 172.21.15.46:0/4098744338 shutdown_connections 2024-04-27T03:54:09.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:09.093+0000 7f248f8ba700 1 -- 172.21.15.46:0/4098744338 wait complete. 2024-04-27T03:54:09.651 INFO:teuthology.orchestra.run.smithi046.stdout:mon is available 2024-04-27T03:54:09.651 INFO:teuthology.orchestra.run.smithi046.stdout:Assimilating anything we can from ceph.conf... 2024-04-27T03:54:09.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:09 smithi046 ceph-mon[26531]: mon.smithi046 is new leader, mons smithi046 in quorum (ranks 0) 2024-04-27T03:54:09.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:09 smithi046 ceph-mon[26531]: monmap e1: 1 mons at {smithi046=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-27T03:54:09.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:09 smithi046 ceph-mon[26531]: fsmap 2024-04-27T03:54:09.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:09 smithi046 ceph-mon[26531]: osdmap e1: 0 total, 0 up, 0 in 2024-04-27T03:54:09.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:09 smithi046 ceph-mon[26531]: mgrmap e1: no daemons active 2024-04-27T03:54:09.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:09 smithi046 ceph-mon[26531]: from='client.? 172.21.15.46:0/4098744338' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-27T03:54:10.704 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-27T03:54:10.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [global] 2024-04-27T03:54:10.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout fsid = b1d3a2d0-0449-11ef-bc93-c7b262605968 2024-04-27T03:54:10.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.46:3300,v1:172.21.15.46:6789] 2024-04-27T03:54:10.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-27T03:54:10.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-27T03:54:10.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-27T03:54:10.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-27T03:54:10.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-27T03:54:10.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-27T03:54:10.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-27T03:54:10.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-27T03:54:10.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [osd] 2024-04-27T03:54:10.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-27T03:54:10.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-27T03:54:10.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-27T03:54:10.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-27T03:54:10.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-27T03:54:10.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.085+0000 7fa994f08700 1 Processor -- start 2024-04-27T03:54:10.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.085+0000 7fa994f08700 1 -- start start 2024-04-27T03:54:10.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.085+0000 7fa994f08700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa99007bea0 0x7fa99007a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:10.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.085+0000 7fa994f08700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99007a8d0 con 0x7fa99007bea0 2024-04-27T03:54:10.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.085+0000 7fa98e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa99007bea0 0x7fa99007a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:10.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.085+0000 7fa98e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa99007bea0 0x7fa99007a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47038/0 (socket says 172.21.15.46:47038) 2024-04-27T03:54:10.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.085+0000 7fa98e59c700 1 -- 172.21.15.46:0/3782553750 learned_addr learned my addr 172.21.15.46:0/3782553750 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:10.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.086+0000 7fa98e59c700 1 -- 172.21.15.46:0/3782553750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99007aa10 con 0x7fa99007bea0 2024-04-27T03:54:10.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.086+0000 7fa98e59c700 1 --2- 172.21.15.46:0/3782553750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa99007bea0 0x7fa99007a300 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa9800098a0 tx=0x7fa980009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b58f32dfc2701c2d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:10.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.086+0000 7fa98dd9b700 1 -- 172.21.15.46:0/3782553750 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa980004180 con 0x7fa99007bea0 2024-04-27T03:54:10.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.086+0000 7fa98dd9b700 1 -- 172.21.15.46:0/3782553750 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9800042e0 con 0x7fa99007bea0 2024-04-27T03:54:10.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.086+0000 7fa98dd9b700 1 -- 172.21.15.46:0/3782553750 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9800036a0 con 0x7fa99007bea0 2024-04-27T03:54:10.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.086+0000 7fa994f08700 1 -- 172.21.15.46:0/3782553750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa99007bea0 msgr2=0x7fa99007a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:10.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.086+0000 7fa994f08700 1 --2- 172.21.15.46:0/3782553750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa99007bea0 0x7fa99007a300 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa9800098a0 tx=0x7fa980009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:10.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.086+0000 7fa994f08700 1 -- 172.21.15.46:0/3782553750 shutdown_connections 2024-04-27T03:54:10.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.086+0000 7fa994f08700 1 --2- 172.21.15.46:0/3782553750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa99007bea0 0x7fa99007a300 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:10.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.086+0000 7fa994f08700 1 -- 172.21.15.46:0/3782553750 >> 172.21.15.46:0/3782553750 conn(0x7fa990101be0 msgr2=0x7fa990104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:10.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.087+0000 7fa994f08700 1 -- 172.21.15.46:0/3782553750 shutdown_connections 2024-04-27T03:54:10.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.087+0000 7fa994f08700 1 -- 172.21.15.46:0/3782553750 wait complete. 2024-04-27T03:54:10.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.087+0000 7fa994f08700 1 Processor -- start 2024-04-27T03:54:10.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.087+0000 7fa994f08700 1 -- start start 2024-04-27T03:54:10.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.088+0000 7fa994f08700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa99007bea0 0x7fa99019bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:10.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.088+0000 7fa994f08700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99019c170 con 0x7fa99007bea0 2024-04-27T03:54:10.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.088+0000 7fa98e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa99007bea0 0x7fa99019bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:10.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.088+0000 7fa98e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa99007bea0 0x7fa99019bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47042/0 (socket says 172.21.15.46:47042) 2024-04-27T03:54:10.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.088+0000 7fa98e59c700 1 -- 172.21.15.46:0/3649742768 learned_addr learned my addr 172.21.15.46:0/3649742768 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:10.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.088+0000 7fa98e59c700 1 -- 172.21.15.46:0/3649742768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa980009550 con 0x7fa99007bea0 2024-04-27T03:54:10.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.088+0000 7fa98e59c700 1 --2- 172.21.15.46:0/3649742768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa99007bea0 0x7fa99019bc30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa99007b370 tx=0x7fa980004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:10.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.088+0000 7fa9877fe700 1 -- 172.21.15.46:0/3649742768 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa980003ce0 con 0x7fa99007bea0 2024-04-27T03:54:10.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.088+0000 7fa9877fe700 1 -- 172.21.15.46:0/3649742768 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa980020070 con 0x7fa99007bea0 2024-04-27T03:54:10.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.088+0000 7fa994f08700 1 -- 172.21.15.46:0/3649742768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99019c370 con 0x7fa99007bea0 2024-04-27T03:54:10.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.088+0000 7fa9877fe700 1 -- 172.21.15.46:0/3649742768 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa980011630 con 0x7fa99007bea0 2024-04-27T03:54:10.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.088+0000 7fa994f08700 1 -- 172.21.15.46:0/3649742768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99019c810 con 0x7fa99007bea0 2024-04-27T03:54:10.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.089+0000 7fa9877fe700 1 -- 172.21.15.46:0/3649742768 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fa980022030 con 0x7fa99007bea0 2024-04-27T03:54:10.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.089+0000 7fa9877fe700 1 -- 172.21.15.46:0/3649742768 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa980025da0 con 0x7fa99007bea0 2024-04-27T03:54:10.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.089+0000 7fa994f08700 1 -- 172.21.15.46:0/3649742768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa990195ba0 con 0x7fa99007bea0 2024-04-27T03:54:10.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.092+0000 7fa9877fe700 1 -- 172.21.15.46:0/3649742768 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fa980025480 con 0x7fa99007bea0 2024-04-27T03:54:10.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.161+0000 7fa994f08700 1 -- 172.21.15.46:0/3649742768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa99007b030 con 0x7fa99007bea0 2024-04-27T03:54:10.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.164+0000 7fa9877fe700 1 -- 172.21.15.46:0/3649742768 <== mon.0 v2:172.21.15.46:3300/0 7 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa980011c20 con 0x7fa99007bea0 2024-04-27T03:54:10.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.164+0000 7fa9877fe700 1 -- 172.21.15.46:0/3649742768 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fa98001c500 con 0x7fa99007bea0 2024-04-27T03:54:10.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.166+0000 7fa994f08700 1 -- 172.21.15.46:0/3649742768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa99007bea0 msgr2=0x7fa99019bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:10.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.166+0000 7fa994f08700 1 --2- 172.21.15.46:0/3649742768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa99007bea0 0x7fa99019bc30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa99007b370 tx=0x7fa980004750 comp rx=0 tx=0).stop 2024-04-27T03:54:10.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.166+0000 7fa994f08700 1 -- 172.21.15.46:0/3649742768 shutdown_connections 2024-04-27T03:54:10.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.166+0000 7fa994f08700 1 --2- 172.21.15.46:0/3649742768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa99007bea0 0x7fa99019bc30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:10.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.166+0000 7fa994f08700 1 -- 172.21.15.46:0/3649742768 >> 172.21.15.46:0/3649742768 conn(0x7fa990101be0 msgr2=0x7fa990102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:10.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.166+0000 7fa994f08700 1 -- 172.21.15.46:0/3649742768 shutdown_connections 2024-04-27T03:54:10.710 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:10.166+0000 7fa994f08700 1 -- 172.21.15.46:0/3649742768 wait complete. 2024-04-27T03:54:10.710 INFO:teuthology.orchestra.run.smithi046.stdout:Generating new minimal ceph.conf... 2024-04-27T03:54:10.814 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:10 smithi046 ceph-mon[26531]: from='client.? 172.21.15.46:0/3649742768' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-27T03:54:10.815 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:10 smithi046 ceph-mon[26531]: from='client.? 172.21.15.46:0/3649742768' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-27T03:54:11.753 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.115+0000 7f7604b8c700 1 Processor -- start 2024-04-27T03:54:11.754 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.115+0000 7f7604b8c700 1 -- start start 2024-04-27T03:54:11.754 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.115+0000 7f7604b8c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76001065c0 0x7f76001069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:11.754 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.115+0000 7f7604b8c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7600106fb0 con 0x7f76001065c0 2024-04-27T03:54:11.754 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.116+0000 7f75fe59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76001065c0 0x7f76001069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:11.754 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.116+0000 7f75fe59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76001065c0 0x7f76001069e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39332/0 (socket says 172.21.15.46:39332) 2024-04-27T03:54:11.754 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.116+0000 7f75fe59c700 1 -- 172.21.15.46:0/236687335 learned_addr learned my addr 172.21.15.46:0/236687335 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:11.754 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.116+0000 7f75fe59c700 1 -- 172.21.15.46:0/236687335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76001077c0 con 0x7f76001065c0 2024-04-27T03:54:11.754 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.116+0000 7f75fe59c700 1 --2- 172.21.15.46:0/236687335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76001065c0 0x7f76001069e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009e80 tx=0x7f75f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ecf3698d11265d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:11.755 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.116+0000 7f75fd59a700 1 -- 172.21.15.46:0/236687335 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75f000e070 con 0x7f76001065c0 2024-04-27T03:54:11.755 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.116+0000 7f75fd59a700 1 -- 172.21.15.46:0/236687335 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f75f000b8c0 con 0x7f76001065c0 2024-04-27T03:54:11.755 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.116+0000 7f75fd59a700 1 -- 172.21.15.46:0/236687335 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75f0004440 con 0x7f76001065c0 2024-04-27T03:54:11.755 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.117+0000 7f7604b8c700 1 -- 172.21.15.46:0/236687335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76001065c0 msgr2=0x7f76001069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:11.755 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.117+0000 7f7604b8c700 1 --2- 172.21.15.46:0/236687335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76001065c0 0x7f76001069e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009e80 tx=0x7f75f000b330 comp rx=0 tx=0).stop 2024-04-27T03:54:11.755 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.117+0000 7f7604b8c700 1 -- 172.21.15.46:0/236687335 shutdown_connections 2024-04-27T03:54:11.755 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.117+0000 7f7604b8c700 1 --2- 172.21.15.46:0/236687335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76001065c0 0x7f76001069e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:11.755 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.117+0000 7f7604b8c700 1 -- 172.21.15.46:0/236687335 >> 172.21.15.46:0/236687335 conn(0x7f7600101be0 msgr2=0x7f7600104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:11.755 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.117+0000 7f7604b8c700 1 -- 172.21.15.46:0/236687335 shutdown_connections 2024-04-27T03:54:11.755 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.117+0000 7f7604b8c700 1 -- 172.21.15.46:0/236687335 wait complete. 2024-04-27T03:54:11.755 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.118+0000 7f7604b8c700 1 Processor -- start 2024-04-27T03:54:11.755 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.118+0000 7f7604b8c700 1 -- start start 2024-04-27T03:54:11.755 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.118+0000 7f7604b8c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76001065c0 0x7f760019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:11.756 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.118+0000 7f7604b8c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760019a820 con 0x7f76001065c0 2024-04-27T03:54:11.756 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.118+0000 7f75fe59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76001065c0 0x7f760019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:11.756 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.118+0000 7f75fe59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76001065c0 0x7f760019a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39348/0 (socket says 172.21.15.46:39348) 2024-04-27T03:54:11.756 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.118+0000 7f75fe59c700 1 -- 172.21.15.46:0/3721420615 learned_addr learned my addr 172.21.15.46:0/3721420615 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:11.756 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.118+0000 7f75fe59c700 1 -- 172.21.15.46:0/3721420615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f0009b30 con 0x7f76001065c0 2024-04-27T03:54:11.756 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.118+0000 7f75fe59c700 1 --2- 172.21.15.46:0/3721420615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76001065c0 0x7f760019a2e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f75f0003c30 tx=0x7f75f0003d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:11.756 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.119+0000 7f75f77fe700 1 -- 172.21.15.46:0/3721420615 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75f000e040 con 0x7f76001065c0 2024-04-27T03:54:11.756 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.119+0000 7f75f77fe700 1 -- 172.21.15.46:0/3721420615 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f75f000bc40 con 0x7f76001065c0 2024-04-27T03:54:11.756 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.119+0000 7f7604b8c700 1 -- 172.21.15.46:0/3721420615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f760019aa20 con 0x7f76001065c0 2024-04-27T03:54:11.756 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.119+0000 7f75f77fe700 1 -- 172.21.15.46:0/3721420615 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75f0012750 con 0x7f76001065c0 2024-04-27T03:54:11.757 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.119+0000 7f7604b8c700 1 -- 172.21.15.46:0/3721420615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f760019af20 con 0x7f76001065c0 2024-04-27T03:54:11.757 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.119+0000 7f75f77fe700 1 -- 172.21.15.46:0/3721420615 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f75f0019040 con 0x7f76001065c0 2024-04-27T03:54:11.757 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.119+0000 7f75f77fe700 1 -- 172.21.15.46:0/3721420615 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f75f0024cf0 con 0x7f76001065c0 2024-04-27T03:54:11.757 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.120+0000 7f7604b8c700 1 -- 172.21.15.46:0/3721420615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7600062510 con 0x7f76001065c0 2024-04-27T03:54:11.757 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.122+0000 7f75f77fe700 1 -- 172.21.15.46:0/3721420615 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f75f001f020 con 0x7f76001065c0 2024-04-27T03:54:11.757 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.192+0000 7f7604b8c700 1 -- 172.21.15.46:0/3721420615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f760010a9f0 con 0x7f76001065c0 2024-04-27T03:54:11.757 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.193+0000 7f75f77fe700 1 -- 172.21.15.46:0/3721420615 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f75f0017070 con 0x7f76001065c0 2024-04-27T03:54:11.757 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.193+0000 7f7604b8c700 1 -- 172.21.15.46:0/3721420615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76001065c0 msgr2=0x7f760019a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:11.757 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.193+0000 7f7604b8c700 1 --2- 172.21.15.46:0/3721420615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76001065c0 0x7f760019a2e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f75f0003c30 tx=0x7f75f0003d10 comp rx=0 tx=0).stop 2024-04-27T03:54:11.757 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.193+0000 7f7604b8c700 1 -- 172.21.15.46:0/3721420615 shutdown_connections 2024-04-27T03:54:11.757 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.193+0000 7f7604b8c700 1 --2- 172.21.15.46:0/3721420615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76001065c0 0x7f760019a2e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:11.757 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.193+0000 7f7604b8c700 1 -- 172.21.15.46:0/3721420615 >> 172.21.15.46:0/3721420615 conn(0x7f7600101be0 msgr2=0x7f7600102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:11.758 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.193+0000 7f7604b8c700 1 -- 172.21.15.46:0/3721420615 shutdown_connections 2024-04-27T03:54:11.758 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:11.193+0000 7f7604b8c700 1 -- 172.21.15.46:0/3721420615 wait complete. 2024-04-27T03:54:11.758 INFO:teuthology.orchestra.run.smithi046.stdout:Restarting the monitor... 2024-04-27T03:54:11.769 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:11 smithi046 ceph-mon[26531]: from='client.? 172.21.15.46:0/3721420615' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:54:12.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:11 smithi046 systemd[1]: Stopping Ceph mon.smithi046 for b1d3a2d0-0449-11ef-bc93-c7b262605968... 2024-04-27T03:54:12.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26527]: 2024-04-27T03:54:12.012+0000 7f8c4489d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi046 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T03:54:12.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26527]: 2024-04-27T03:54:12.012+0000 7f8c4489d700 -1 mon.smithi046@0(leader) e1 *** Got Signal Terminated *** 2024-04-27T03:54:12.465 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:12 smithi046 podman[26773]: 2024-04-27 03:54:12.197415618 +0000 UTC m=+0.284489269 container died 603e44932c68e1c52c2daf5ed8e6af197f7e1efe91a275dbbf817ed28f38ee53 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=reef-5403a02, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-27T03:54:12.898 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:12 smithi046 podman[26773]: 2024-04-27 03:54:12.537177045 +0000 UTC m=+0.624250696 container cleanup 603e44932c68e1c52c2daf5ed8e6af197f7e1efe91a275dbbf817ed28f38ee53 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=reef-5403a02, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-27T03:54:12.898 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:12 smithi046 bash[26773]: ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046 2024-04-27T03:54:12.898 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:12 smithi046 podman[26787]: 2024-04-27 03:54:12.679908263 +0000 UTC m=+0.483495426 container remove 603e44932c68e1c52c2daf5ed8e6af197f7e1efe91a275dbbf817ed28f38ee53 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=reef-5403a02, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-27T03:54:13.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:12 smithi046 systemd[1]: ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@mon.smithi046.service: Deactivated successfully. 2024-04-27T03:54:13.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:12 smithi046 systemd[1]: Stopped Ceph mon.smithi046 for b1d3a2d0-0449-11ef-bc93-c7b262605968. 2024-04-27T03:54:13.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:12 smithi046 systemd[1]: Starting Ceph mon.smithi046 for b1d3a2d0-0449-11ef-bc93-c7b262605968... 2024-04-27T03:54:13.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 podman[26880]: 2024-04-27 03:54:13.114108448 +0000 UTC m=+0.105230408 container create 72fac097effc4f80a0d126f7d48ff847c476190bc241aab5ea0cb6aaa8298132 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-5403a02, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-27T03:54:13.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 podman[26880]: 2024-04-27 03:54:13.02718636 +0000 UTC m=+0.018308298 image pull 267a473d505df5b518068223eb9431973cc4aab49462dd79c898418ebe6d341a quay.ceph.io/ceph-ci/ceph:reef 2024-04-27T03:54:13.390 INFO:teuthology.orchestra.run.smithi046.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-27T03:54:13.512 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 podman[26880]: 2024-04-27 03:54:13.32461028 +0000 UTC m=+0.315732204 container init 72fac097effc4f80a0d126f7d48ff847c476190bc241aab5ea0cb6aaa8298132 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=reef-5403a02, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T03:54:13.512 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 podman[26880]: 2024-04-27 03:54:13.327494087 +0000 UTC m=+0.318616010 container start 72fac097effc4f80a0d126f7d48ff847c476190bc241aab5ea0cb6aaa8298132 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=reef-5403a02, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-04-27T03:54:13.512 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: set uid:gid to 167:167 (ceph:ceph) 2024-04-27T03:54:13.512 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable), process ceph-mon, pid 2 2024-04-27T03:54:13.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: pidfile_write: ignore empty --pid-file 2024-04-27T03:54:13.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: load: jerasure load: lrc 2024-04-27T03:54:13.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: RocksDB version: 7.9.2 2024-04-27T03:54:13.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Git sha 0 2024-04-27T03:54:13.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Compile date 2024-04-26 15:24:09 2024-04-27T03:54:13.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: DB SUMMARY 2024-04-27T03:54:13.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: DB Session ID: 3947MSV0F7RUMQLXGEO4 2024-04-27T03:54:13.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: CURRENT file: CURRENT 2024-04-27T03:54:13.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: IDENTITY file: IDENTITY 2024-04-27T03:54:13.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-27T03:54:13.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi046/store.db dir, Total Num: 1, files: 000008.sst 2024-04-27T03:54:13.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi046/store.db: 000009.log size: 98487 ; 2024-04-27T03:54:13.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.error_if_exists: 0 2024-04-27T03:54:13.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.create_if_missing: 0 2024-04-27T03:54:13.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.paranoid_checks: 1 2024-04-27T03:54:13.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-27T03:54:13.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-27T03:54:13.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-27T03:54:13.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.env: 0x55cf5ca4d820 2024-04-27T03:54:13.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.fs: PosixFileSystem 2024-04-27T03:54:13.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.info_log: 0x55cf5e8313c0 2024-04-27T03:54:13.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_file_opening_threads: 16 2024-04-27T03:54:13.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.statistics: (nil) 2024-04-27T03:54:13.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.use_fsync: 0 2024-04-27T03:54:13.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_log_file_size: 0 2024-04-27T03:54:13.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-27T03:54:13.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-27T03:54:13.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.keep_log_file_num: 1000 2024-04-27T03:54:13.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.recycle_log_file_num: 0 2024-04-27T03:54:13.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.allow_fallocate: 1 2024-04-27T03:54:13.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.allow_mmap_reads: 0 2024-04-27T03:54:13.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.allow_mmap_writes: 0 2024-04-27T03:54:13.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.use_direct_reads: 0 2024-04-27T03:54:13.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-27T03:54:13.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.create_missing_column_families: 0 2024-04-27T03:54:13.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.db_log_dir: 2024-04-27T03:54:13.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.wal_dir: 2024-04-27T03:54:13.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-27T03:54:13.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-27T03:54:13.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-27T03:54:13.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-27T03:54:13.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-27T03:54:13.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-27T03:54:13.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.advise_random_on_open: 1 2024-04-27T03:54:13.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.db_write_buffer_size: 0 2024-04-27T03:54:13.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.write_buffer_manager: 0x55cf5daa6140 2024-04-27T03:54:13.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-27T03:54:13.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-27T03:54:13.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-27T03:54:13.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.rate_limiter: (nil) 2024-04-27T03:54:13.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-27T03:54:13.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.wal_recovery_mode: 2 2024-04-27T03:54:13.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.enable_thread_tracking: 0 2024-04-27T03:54:13.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.enable_pipelined_write: 0 2024-04-27T03:54:13.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.unordered_write: 0 2024-04-27T03:54:13.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-27T03:54:13.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-27T03:54:13.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-27T03:54:13.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-27T03:54:13.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.row_cache: None 2024-04-27T03:54:13.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.wal_filter: None 2024-04-27T03:54:13.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-27T03:54:13.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.allow_ingest_behind: 0 2024-04-27T03:54:13.518 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.two_write_queues: 0 2024-04-27T03:54:13.518 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.manual_wal_flush: 0 2024-04-27T03:54:13.519 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.wal_compression: 0 2024-04-27T03:54:13.519 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.atomic_flush: 0 2024-04-27T03:54:13.519 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-27T03:54:13.519 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-27T03:54:13.519 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-27T03:54:13.519 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.log_readahead_size: 0 2024-04-27T03:54:13.519 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-27T03:54:13.519 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.best_efforts_recovery: 0 2024-04-27T03:54:13.519 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-27T03:54:13.519 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-27T03:54:13.519 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.allow_data_in_errors: 0 2024-04-27T03:54:13.519 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.db_host_id: __hostname__ 2024-04-27T03:54:13.519 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-27T03:54:13.519 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_background_jobs: 2 2024-04-27T03:54:13.520 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_background_compactions: -1 2024-04-27T03:54:13.520 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_subcompactions: 1 2024-04-27T03:54:13.520 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-27T03:54:13.520 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-27T03:54:13.520 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-27T03:54:13.520 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_total_wal_size: 0 2024-04-27T03:54:13.520 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-27T03:54:13.520 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-27T03:54:13.520 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-27T03:54:13.520 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-27T03:54:13.520 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_open_files: -1 2024-04-27T03:54:13.520 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.bytes_per_sync: 0 2024-04-27T03:54:13.520 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-27T03:54:13.521 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-27T03:54:13.521 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compaction_readahead_size: 0 2024-04-27T03:54:13.521 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_background_flushes: -1 2024-04-27T03:54:13.521 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Compression algorithms supported: 2024-04-27T03:54:13.521 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-27T03:54:13.521 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: kZSTD supported: 0 2024-04-27T03:54:13.521 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: kXpressCompression supported: 0 2024-04-27T03:54:13.521 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: kLZ4HCCompression supported: 1 2024-04-27T03:54:13.521 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: kZlibCompression supported: 1 2024-04-27T03:54:13.521 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: kSnappyCompression supported: 1 2024-04-27T03:54:13.521 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: kLZ4Compression supported: 1 2024-04-27T03:54:13.521 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: kBZip2Compression supported: 0 2024-04-27T03:54:13.521 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-27T03:54:13.521 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-27T03:54:13.522 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi046/store.db/MANIFEST-000010 2024-04-27T03:54:13.522 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-27T03:54:13.522 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-27T03:54:13.522 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.merge_operator: 2024-04-27T03:54:13.522 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compaction_filter: None 2024-04-27T03:54:13.522 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compaction_filter_factory: None 2024-04-27T03:54:13.524 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.sst_partitioner_factory: None 2024-04-27T03:54:13.524 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-27T03:54:13.524 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-27T03:54:13.524 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cf5e8314e0) 2024-04-27T03:54:13.524 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: cache_index_and_filter_blocks: 1 2024-04-27T03:54:13.524 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-27T03:54:13.524 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-27T03:54:13.524 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: pin_top_level_index_and_filter: 1 2024-04-27T03:54:13.524 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: index_type: 0 2024-04-27T03:54:13.524 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: data_block_index_type: 0 2024-04-27T03:54:13.524 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: index_shortening: 1 2024-04-27T03:54:13.524 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-27T03:54:13.524 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: checksum: 4 2024-04-27T03:54:13.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: no_block_cache: 0 2024-04-27T03:54:13.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: block_cache: 0x55cf5db471f0 2024-04-27T03:54:13.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: block_cache_name: BinnedLRUCache 2024-04-27T03:54:13.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: block_cache_options: 2024-04-27T03:54:13.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: capacity : 536870912 2024-04-27T03:54:13.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: num_shard_bits : 4 2024-04-27T03:54:13.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: strict_capacity_limit : 0 2024-04-27T03:54:13.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: high_pri_pool_ratio: 0.000 2024-04-27T03:54:13.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: block_cache_compressed: (nil) 2024-04-27T03:54:13.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: persistent_cache: (nil) 2024-04-27T03:54:13.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: block_size: 4096 2024-04-27T03:54:13.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: block_size_deviation: 10 2024-04-27T03:54:13.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: block_restart_interval: 16 2024-04-27T03:54:13.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: index_block_restart_interval: 1 2024-04-27T03:54:13.526 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: metadata_block_size: 4096 2024-04-27T03:54:13.526 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: partition_filters: 0 2024-04-27T03:54:13.526 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: use_delta_encoding: 1 2024-04-27T03:54:13.526 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: filter_policy: bloomfilter 2024-04-27T03:54:13.526 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: whole_key_filtering: 1 2024-04-27T03:54:13.526 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: verify_compression: 0 2024-04-27T03:54:13.526 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: read_amp_bytes_per_bit: 0 2024-04-27T03:54:13.526 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: format_version: 5 2024-04-27T03:54:13.526 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: enable_index_compression: 1 2024-04-27T03:54:13.526 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: block_align: 0 2024-04-27T03:54:13.526 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: max_auto_readahead_size: 262144 2024-04-27T03:54:13.526 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: prepopulate_block_cache: 0 2024-04-27T03:54:13.526 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: initial_auto_readahead_size: 8192 2024-04-27T03:54:13.526 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: num_file_reads_for_auto_readahead: 2 2024-04-27T03:54:13.527 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.write_buffer_size: 33554432 2024-04-27T03:54:13.527 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_write_buffer_number: 2 2024-04-27T03:54:13.527 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compression: NoCompression 2024-04-27T03:54:13.527 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.bottommost_compression: Disabled 2024-04-27T03:54:13.527 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.prefix_extractor: nullptr 2024-04-27T03:54:13.527 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-27T03:54:13.527 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.num_levels: 7 2024-04-27T03:54:13.527 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-27T03:54:13.527 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-27T03:54:13.527 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-27T03:54:13.527 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-27T03:54:13.527 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-27T03:54:13.527 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-27T03:54:13.527 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-27T03:54:13.528 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-27T03:54:13.528 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-27T03:54:13.528 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-27T03:54:13.528 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-27T03:54:13.528 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-27T03:54:13.528 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-27T03:54:13.528 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compression_opts.level: 32767 2024-04-27T03:54:13.528 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compression_opts.strategy: 0 2024-04-27T03:54:13.528 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-27T03:54:13.528 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-27T03:54:13.528 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-27T03:54:13.528 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-27T03:54:13.528 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compression_opts.enabled: false 2024-04-27T03:54:13.528 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-27T03:54:13.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-27T03:54:13.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-27T03:54:13.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-27T03:54:13.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.target_file_size_base: 67108864 2024-04-27T03:54:13.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-27T03:54:13.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-27T03:54:13.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-27T03:54:13.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-27T03:54:13.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-27T03:54:13.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-27T03:54:13.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-27T03:54:13.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-27T03:54:13.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-27T03:54:13.530 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-27T03:54:13.530 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-27T03:54:13.530 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-27T03:54:13.530 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-27T03:54:13.530 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-27T03:54:13.530 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.arena_block_size: 1048576 2024-04-27T03:54:13.530 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-27T03:54:13.530 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-27T03:54:13.530 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.disable_auto_compactions: 0 2024-04-27T03:54:13.530 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-27T03:54:13.531 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-27T03:54:13.531 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-27T03:54:13.531 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-27T03:54:13.531 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-27T03:54:13.531 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-27T03:54:13.531 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-27T03:54:13.531 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-27T03:54:13.531 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-27T03:54:13.531 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-27T03:54:13.531 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.table_properties_collectors: 2024-04-27T03:54:13.531 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.inplace_update_support: 0 2024-04-27T03:54:13.531 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-27T03:54:13.531 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-27T03:54:13.531 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-27T03:54:13.532 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-27T03:54:13.532 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.bloom_locality: 0 2024-04-27T03:54:13.532 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.max_successive_merges: 0 2024-04-27T03:54:13.532 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-27T03:54:13.532 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.paranoid_file_checks: 0 2024-04-27T03:54:13.532 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.force_consistency_checks: 1 2024-04-27T03:54:13.532 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.report_bg_io_stats: 0 2024-04-27T03:54:13.532 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.ttl: 2592000 2024-04-27T03:54:13.532 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-27T03:54:13.532 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-27T03:54:13.532 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-27T03:54:13.532 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.enable_blob_files: false 2024-04-27T03:54:13.532 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.min_blob_size: 0 2024-04-27T03:54:13.532 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.blob_file_size: 268435456 2024-04-27T03:54:13.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-27T03:54:13.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-27T03:54:13.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-27T03:54:13.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-27T03:54:13.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-27T03:54:13.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.blob_file_starting_level: 0 2024-04-27T03:54:13.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-27T03:54:13.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi046/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-27T03:54:13.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-27T03:54:13.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 83235276-e886-4170-828d-50bc4dde1dc5 2024-04-27T03:54:13.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714190053359027, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-27T03:54:13.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-27T03:54:13.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714190053360100, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93764, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91798, "index_size": 248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84362, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714190053, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "83235276-e886-4170-828d-50bc4dde1dc5", "db_session_id": "3947MSV0F7RUMQLXGEO4", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-27T03:54:13.534 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714190053360199, "job": 1, "event": "recovery_finished"} 2024-04-27T03:54:13.534 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-27T03:54:13.534 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi046/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-27T03:54:13.534 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55cf5dbe6000 2024-04-27T03:54:13.534 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: DB pointer 0x55cf5dbd4000 2024-04-27T03:54:13.534 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-27T03:54:13.534 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-27T03:54:13.534 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: ** DB Stats ** 2024-04-27T03:54:13.534 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T03:54:13.534 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-27T03:54:13.534 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T03:54:13.534 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T03:54:13.534 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-27T03:54:13.534 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T03:54:13.535 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T03:54:13.535 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: 2024-04-27T03:54:13.535 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: ** Compaction Stats [default] ** 2024-04-27T03:54:13.535 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T03:54:13.535 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-27T03:54:13.535 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: L0 2/0 93.44 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 132.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T03:54:13.535 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Sum 2/0 93.44 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 132.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T03:54:13.535 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 132.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T03:54:13.535 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: 2024-04-27T03:54:13.535 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: ** Compaction Stats [default] ** 2024-04-27T03:54:13.535 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T03:54:13.535 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T03:54:13.535 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 132.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T03:54:13.535 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: 2024-04-27T03:54:13.536 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-27T03:54:13.536 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: 2024-04-27T03:54:13.536 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T03:54:13.536 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-27T03:54:13.536 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-27T03:54:13.536 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-27T03:54:13.536 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-27T03:54:13.536 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-27T03:54:13.536 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Cumulative compaction: 0.00 GB write, 29.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T03:54:13.536 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Interval compaction: 0.00 GB write, 29.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T03:54:13.536 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-27T03:54:13.536 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Block cache BinnedLRUCache@0x55cf5db471f0#2 capacity: 512.00 MB usage: 1.33 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-27T03:54:13.536 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-04-27T03:54:13.536 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: 2024-04-27T03:54:13.537 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-27T03:54:13.537 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: starting mon.smithi046 rank 0 at public addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] at bind addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi046 fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 2024-04-27T03:54:13.537 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: mon.smithi046@-1(???) e1 preinit fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 2024-04-27T03:54:13.537 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: mon.smithi046@-1(???).mds e1 new map 2024-04-27T03:54:13.537 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: mon.smithi046@-1(???).mds e1 print_map 2024-04-27T03:54:13.537 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: e1 2024-04-27T03:54:13.537 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T03:54:13.537 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T03:54:13.537 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: legacy client fscid: -1 2024-04-27T03:54:13.537 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: 2024-04-27T03:54:13.537 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout: No filesystems configured 2024-04-27T03:54:13.537 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: mon.smithi046@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-27T03:54:13.537 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: mon.smithi046@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T03:54:13.537 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: mon.smithi046@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T03:54:13.537 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: mon.smithi046@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T03:54:13.538 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: mon.smithi046@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-27T03:54:13.538 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 bash[26880]: 72fac097effc4f80a0d126f7d48ff847c476190bc241aab5ea0cb6aaa8298132 2024-04-27T03:54:13.538 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 systemd[1]: Started Ceph mon.smithi046 for b1d3a2d0-0449-11ef-bc93-c7b262605968. 2024-04-27T03:54:13.538 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: mon.smithi046 is new leader, mons smithi046 in quorum (ranks 0) 2024-04-27T03:54:13.538 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: monmap e1: 1 mons at {smithi046=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-27T03:54:13.538 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: fsmap 2024-04-27T03:54:13.538 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: osdmap e1: 0 total, 0 up, 0 in 2024-04-27T03:54:13.538 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:13 smithi046 ceph-mon[26899]: mgrmap e1: no daemons active 2024-04-27T03:54:14.474 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.883+0000 7f5df08b8700 1 Processor -- start 2024-04-27T03:54:14.474 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.883+0000 7f5df08b8700 1 -- start start 2024-04-27T03:54:14.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.883+0000 7f5df08b8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dec107030 0x7f5dec109420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:14.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.883+0000 7f5df08b8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec079ed0 con 0x7f5dec107030 2024-04-27T03:54:14.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.883+0000 7f5dea59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dec107030 0x7f5dec109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:14.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.883+0000 7f5dea59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dec107030 0x7f5dec109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39352/0 (socket says 172.21.15.46:39352) 2024-04-27T03:54:14.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.883+0000 7f5dea59c700 1 -- 172.21.15.46:0/2327617042 learned_addr learned my addr 172.21.15.46:0/2327617042 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:14.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.884+0000 7f5dea59c700 1 -- 172.21.15.46:0/2327617042 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec109960 con 0x7f5dec107030 2024-04-27T03:54:14.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.884+0000 7f5dea59c700 1 --2- 172.21.15.46:0/2327617042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dec107030 0x7f5dec109420 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc0098a0 tx=0x7f5ddc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eefe2cbb268618b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:14.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.884+0000 7f5de959a700 1 -- 172.21.15.46:0/2327617042 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ddc004180 con 0x7f5dec107030 2024-04-27T03:54:14.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.884+0000 7f5de959a700 1 -- 172.21.15.46:0/2327617042 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5ddc0042e0 con 0x7f5dec107030 2024-04-27T03:54:14.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.884+0000 7f5de959a700 1 -- 172.21.15.46:0/2327617042 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ddc011a70 con 0x7f5dec107030 2024-04-27T03:54:14.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.884+0000 7f5df08b8700 1 -- 172.21.15.46:0/2327617042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dec107030 msgr2=0x7f5dec109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:14.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.884+0000 7f5df08b8700 1 --2- 172.21.15.46:0/2327617042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dec107030 0x7f5dec109420 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc0098a0 tx=0x7f5ddc009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:14.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.885+0000 7f5df08b8700 1 -- 172.21.15.46:0/2327617042 shutdown_connections 2024-04-27T03:54:14.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.885+0000 7f5df08b8700 1 --2- 172.21.15.46:0/2327617042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dec107030 0x7f5dec109420 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:14.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.885+0000 7f5df08b8700 1 -- 172.21.15.46:0/2327617042 >> 172.21.15.46:0/2327617042 conn(0x7f5dec100ae0 msgr2=0x7f5dec102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:14.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.885+0000 7f5df08b8700 1 -- 172.21.15.46:0/2327617042 shutdown_connections 2024-04-27T03:54:14.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.885+0000 7f5df08b8700 1 -- 172.21.15.46:0/2327617042 wait complete. 2024-04-27T03:54:14.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.885+0000 7f5df08b8700 1 Processor -- start 2024-04-27T03:54:14.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.885+0000 7f5df08b8700 1 -- start start 2024-04-27T03:54:14.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.886+0000 7f5df08b8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dec107030 0x7f5dec19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:14.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.886+0000 7f5df08b8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec19a7d0 con 0x7f5dec107030 2024-04-27T03:54:14.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.886+0000 7f5dea59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dec107030 0x7f5dec19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:14.477 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.886+0000 7f5dea59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dec107030 0x7f5dec19a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39358/0 (socket says 172.21.15.46:39358) 2024-04-27T03:54:14.477 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.886+0000 7f5dea59c700 1 -- 172.21.15.46:0/978929453 learned_addr learned my addr 172.21.15.46:0/978929453 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:14.477 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.886+0000 7f5dea59c700 1 -- 172.21.15.46:0/978929453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ddc009550 con 0x7f5dec107030 2024-04-27T03:54:14.477 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.886+0000 7f5dea59c700 1 --2- 172.21.15.46:0/978929453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dec107030 0x7f5dec19a290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc011a20 tx=0x7f5ddc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:14.477 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.886+0000 7f5ddb7fe700 1 -- 172.21.15.46:0/978929453 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ddc003b10 con 0x7f5dec107030 2024-04-27T03:54:14.477 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.886+0000 7f5ddb7fe700 1 -- 172.21.15.46:0/978929453 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5ddc01b8c0 con 0x7f5dec107030 2024-04-27T03:54:14.477 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.886+0000 7f5df08b8700 1 -- 172.21.15.46:0/978929453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dec19a9d0 con 0x7f5dec107030 2024-04-27T03:54:14.477 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.886+0000 7f5ddb7fe700 1 -- 172.21.15.46:0/978929453 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ddc0237b0 con 0x7f5dec107030 2024-04-27T03:54:14.477 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.886+0000 7f5df08b8700 1 -- 172.21.15.46:0/978929453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dec19ae70 con 0x7f5dec107030 2024-04-27T03:54:14.477 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.887+0000 7f5ddb7fe700 1 -- 172.21.15.46:0/978929453 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f5ddc023d30 con 0x7f5dec107030 2024-04-27T03:54:14.477 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.887+0000 7f5ddb7fe700 1 -- 172.21.15.46:0/978929453 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5ddc019710 con 0x7f5dec107030 2024-04-27T03:54:14.477 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.887+0000 7f5df08b8700 1 -- 172.21.15.46:0/978929453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dec10a6e0 con 0x7f5dec107030 2024-04-27T03:54:14.477 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.890+0000 7f5ddb7fe700 1 -- 172.21.15.46:0/978929453 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f5ddc01ba30 con 0x7f5dec107030 2024-04-27T03:54:14.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.960+0000 7f5df08b8700 1 -- 172.21.15.46:0/978929453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f5dec10a8d0 con 0x7f5dec107030 2024-04-27T03:54:14.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.962+0000 7f5ddb7fe700 1 -- 172.21.15.46:0/978929453 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5ddc01bbd0 con 0x7f5dec107030 2024-04-27T03:54:14.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.963+0000 7f5df08b8700 1 -- 172.21.15.46:0/978929453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dec107030 msgr2=0x7f5dec19a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:14.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.963+0000 7f5df08b8700 1 --2- 172.21.15.46:0/978929453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dec107030 0x7f5dec19a290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc011a20 tx=0x7f5ddc0047c0 comp rx=0 tx=0).stop 2024-04-27T03:54:14.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.963+0000 7f5df08b8700 1 -- 172.21.15.46:0/978929453 shutdown_connections 2024-04-27T03:54:14.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.963+0000 7f5df08b8700 1 --2- 172.21.15.46:0/978929453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dec107030 0x7f5dec19a290 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:14.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.963+0000 7f5df08b8700 1 -- 172.21.15.46:0/978929453 >> 172.21.15.46:0/978929453 conn(0x7f5dec100ae0 msgr2=0x7f5dec1015b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:14.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.963+0000 7f5df08b8700 1 -- 172.21.15.46:0/978929453 shutdown_connections 2024-04-27T03:54:14.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:13.963+0000 7f5df08b8700 1 -- 172.21.15.46:0/978929453 wait complete. 2024-04-27T03:54:14.478 INFO:teuthology.orchestra.run.smithi046.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-27T03:54:14.492 INFO:teuthology.orchestra.run.smithi046.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-27T03:54:14.492 INFO:teuthology.orchestra.run.smithi046.stdout:Creating mgr... 2024-04-27T03:54:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-27T03:54:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-27T03:54:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying port 0.0.0.0:8443 ... 2024-04-27T03:54:14.740 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@mgr.smithi046.qrfdvr 2024-04-27T03:54:14.758 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@mgr.smithi046.qrfdvr.service: Unit ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@mgr.smithi046.qrfdvr.service not loaded. 2024-04-27T03:54:14.896 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968.target.wants/ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@mgr.smithi046.qrfdvr.service → /etc/systemd/system/ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@.service. 2024-04-27T03:54:15.129 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:14 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/978929453' entity='client.admin' 2024-04-27T03:54:15.422 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-27T03:54:15.422 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2024-04-27T03:54:15.430 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-27T03:54:15.430 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-04-27T03:54:15.430 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2024-04-27T03:54:15.431 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to enable service . firewalld.service is not available 2024-04-27T03:54:15.441 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-27T03:54:15.441 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2024-04-27T03:54:15.450 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-27T03:54:15.450 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-04-27T03:54:15.450 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2024-04-27T03:54:15.451 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-04-27T03:54:15.451 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr to start... 2024-04-27T03:54:15.451 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr... 2024-04-27T03:54:16.232 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:16 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2857528891' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T03:54:16.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-27T03:54:16.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-04-27T03:54:16.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "b1d3a2d0-0449-11ef-bc93-c7b262605968", 2024-04-27T03:54:16.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2024-04-27T03:54:16.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-27T03:54:16.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-27T03:54:16.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-27T03:54:16.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:16.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-27T03:54:16.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-27T03:54:16.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2024-04-27T03:54:16.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-27T03:54:16.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-27T03:54:16.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "smithi046" 2024-04-27T03:54:16.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-27T03:54:16.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-27T03:54:16.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-27T03:54:16.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T03:54:16.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-27T03:54:16.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-27T03:54:16.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:16.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-27T03:54:16.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T03:54:16.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-27T03:54:16.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-27T03:54:16.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-27T03:54:16.527 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-27T03:54:16.527 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-27T03:54:16.527 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-27T03:54:16.527 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:16.527 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-27T03:54:16.528 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-27T03:54:16.528 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-27T03:54:16.528 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-27T03:54:16.528 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-27T03:54:16.528 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-27T03:54:16.528 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-27T03:54:16.528 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-27T03:54:16.528 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-27T03:54:16.528 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:16.528 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-27T03:54:16.528 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T03:54:16.528 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-27T03:54:16.528 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-27T03:54:16.528 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:16.529 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-27T03:54:16.529 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-27T03:54:16.529 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-27T03:54:16.529 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-27T03:54:16.529 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-27T03:54:16.529 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-27T03:54:16.529 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2024-04-27T03:54:16.529 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-27T03:54:16.529 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T03:54:16.529 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:16.529 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-27T03:54:16.529 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T03:54:16.529 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2024-04-27T03:54:08.503101+0000", 2024-04-27T03:54:16.529 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T03:54:16.530 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:16.530 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-27T03:54:16.530 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-04-27T03:54:16.530 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.892+0000 7fa39084f700 1 Processor -- start 2024-04-27T03:54:16.530 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.892+0000 7fa39084f700 1 -- start start 2024-04-27T03:54:16.530 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.892+0000 7fa39084f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c1048b0 0x7fa38c104cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:16.530 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.892+0000 7fa39084f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c1052a0 con 0x7fa38c1048b0 2024-04-27T03:54:16.530 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.893+0000 7fa38a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c1048b0 0x7fa38c104cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:16.530 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.893+0000 7fa38a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c1048b0 0x7fa38c104cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39382/0 (socket says 172.21.15.46:39382) 2024-04-27T03:54:16.530 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.893+0000 7fa38a59c700 1 -- 172.21.15.46:0/1284723590 learned_addr learned my addr 172.21.15.46:0/1284723590 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:16.530 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.893+0000 7fa38a59c700 1 -- 172.21.15.46:0/1284723590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c107840 con 0x7fa38c1048b0 2024-04-27T03:54:16.530 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.893+0000 7fa38a59c700 1 --2- 172.21.15.46:0/1284723590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c1048b0 0x7fa38c104cd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa37c0098a0 tx=0x7fa37c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2633ec141cd12444 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:16.530 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.893+0000 7fa38959a700 1 -- 172.21.15.46:0/1284723590 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37c004180 con 0x7fa38c1048b0 2024-04-27T03:54:16.531 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.893+0000 7fa38959a700 1 -- 172.21.15.46:0/1284723590 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa37c0042e0 con 0x7fa38c1048b0 2024-04-27T03:54:16.531 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.893+0000 7fa38959a700 1 -- 172.21.15.46:0/1284723590 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37c011a70 con 0x7fa38c1048b0 2024-04-27T03:54:16.531 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.894+0000 7fa39084f700 1 -- 172.21.15.46:0/1284723590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c1048b0 msgr2=0x7fa38c104cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:16.531 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.894+0000 7fa39084f700 1 --2- 172.21.15.46:0/1284723590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c1048b0 0x7fa38c104cd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa37c0098a0 tx=0x7fa37c009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:16.531 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.894+0000 7fa39084f700 1 -- 172.21.15.46:0/1284723590 shutdown_connections 2024-04-27T03:54:16.531 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.894+0000 7fa39084f700 1 --2- 172.21.15.46:0/1284723590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c1048b0 0x7fa38c104cd0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:16.531 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.894+0000 7fa39084f700 1 -- 172.21.15.46:0/1284723590 >> 172.21.15.46:0/1284723590 conn(0x7fa38c1001c0 msgr2=0x7fa38c102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:16.531 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.894+0000 7fa39084f700 1 -- 172.21.15.46:0/1284723590 shutdown_connections 2024-04-27T03:54:16.531 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.894+0000 7fa39084f700 1 -- 172.21.15.46:0/1284723590 wait complete. 2024-04-27T03:54:16.531 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.895+0000 7fa39084f700 1 Processor -- start 2024-04-27T03:54:16.531 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.895+0000 7fa39084f700 1 -- start start 2024-04-27T03:54:16.531 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.895+0000 7fa39084f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c1048b0 0x7fa38c19a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:16.531 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.895+0000 7fa39084f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c19a8b0 con 0x7fa38c1048b0 2024-04-27T03:54:16.532 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.895+0000 7fa38a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c1048b0 0x7fa38c19a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:16.532 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.895+0000 7fa38a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c1048b0 0x7fa38c19a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39386/0 (socket says 172.21.15.46:39386) 2024-04-27T03:54:16.532 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.895+0000 7fa38a59c700 1 -- 172.21.15.46:0/2857528891 learned_addr learned my addr 172.21.15.46:0/2857528891 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:16.532 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.896+0000 7fa38a59c700 1 -- 172.21.15.46:0/2857528891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c009550 con 0x7fa38c1048b0 2024-04-27T03:54:16.532 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.896+0000 7fa38a59c700 1 --2- 172.21.15.46:0/2857528891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c1048b0 0x7fa38c19a370 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa37c011710 tx=0x7fa37c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:16.532 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.896+0000 7fa37b7fe700 1 -- 172.21.15.46:0/2857528891 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37c003b10 con 0x7fa38c1048b0 2024-04-27T03:54:16.532 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.896+0000 7fa39084f700 1 -- 172.21.15.46:0/2857528891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa38c19aab0 con 0x7fa38c1048b0 2024-04-27T03:54:16.532 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.896+0000 7fa37b7fe700 1 -- 172.21.15.46:0/2857528891 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa37c01b8c0 con 0x7fa38c1048b0 2024-04-27T03:54:16.532 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.896+0000 7fa37b7fe700 1 -- 172.21.15.46:0/2857528891 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37c019770 con 0x7fa38c1048b0 2024-04-27T03:54:16.532 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.896+0000 7fa39084f700 1 -- 172.21.15.46:0/2857528891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38c19af50 con 0x7fa38c1048b0 2024-04-27T03:54:16.532 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.896+0000 7fa37b7fe700 1 -- 172.21.15.46:0/2857528891 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fa37c019cf0 con 0x7fa38c1048b0 2024-04-27T03:54:16.532 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.896+0000 7fa37b7fe700 1 -- 172.21.15.46:0/2857528891 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa37c01a7e0 con 0x7fa38c1048b0 2024-04-27T03:54:16.532 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.897+0000 7fa39084f700 1 -- 172.21.15.46:0/2857528891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38c062510 con 0x7fa38c1048b0 2024-04-27T03:54:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.901+0000 7fa37b7fe700 1 -- 172.21.15.46:0/2857528891 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fa37c029060 con 0x7fa38c1048b0 2024-04-27T03:54:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.975+0000 7fa39084f700 1 -- 172.21.15.46:0/2857528891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa38c19db00 con 0x7fa38c1048b0 2024-04-27T03:54:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.975+0000 7fa37b7fe700 1 -- 172.21.15.46:0/2857528891 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fa37c01f070 con 0x7fa38c1048b0 2024-04-27T03:54:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.977+0000 7fa39084f700 1 -- 172.21.15.46:0/2857528891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c1048b0 msgr2=0x7fa38c19a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.977+0000 7fa39084f700 1 --2- 172.21.15.46:0/2857528891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c1048b0 0x7fa38c19a370 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa37c011710 tx=0x7fa37c0047c0 comp rx=0 tx=0).stop 2024-04-27T03:54:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.977+0000 7fa39084f700 1 -- 172.21.15.46:0/2857528891 shutdown_connections 2024-04-27T03:54:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.977+0000 7fa39084f700 1 --2- 172.21.15.46:0/2857528891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c1048b0 0x7fa38c19a370 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.977+0000 7fa39084f700 1 -- 172.21.15.46:0/2857528891 >> 172.21.15.46:0/2857528891 conn(0x7fa38c1001c0 msgr2=0x7fa38c100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.977+0000 7fa39084f700 1 -- 172.21.15.46:0/2857528891 shutdown_connections 2024-04-27T03:54:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:15.977+0000 7fa39084f700 1 -- 172.21.15.46:0/2857528891 wait complete. 2024-04-27T03:54:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:mgr not available, waiting (1/15)... 2024-04-27T03:54:21.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:21 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/587856933' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T03:54:23.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:22 smithi046 ceph-mon[26899]: Activating manager daemon smithi046.qrfdvr 2024-04-27T03:54:23.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:22 smithi046 ceph-mon[26899]: mgrmap e2: smithi046.qrfdvr(active, starting, since 0.00327402s) 2024-04-27T03:54:23.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:22 smithi046 ceph-mon[26899]: from='mgr.14100 172.21.15.46:0/2302581870' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T03:54:23.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:22 smithi046 ceph-mon[26899]: from='mgr.14100 172.21.15.46:0/2302581870' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T03:54:23.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:22 smithi046 ceph-mon[26899]: from='mgr.14100 172.21.15.46:0/2302581870' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T03:54:23.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:22 smithi046 ceph-mon[26899]: from='mgr.14100 172.21.15.46:0/2302581870' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-04-27T03:54:23.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:22 smithi046 ceph-mon[26899]: from='mgr.14100 172.21.15.46:0/2302581870' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mgr metadata", "who": "smithi046.qrfdvr", "id": "smithi046.qrfdvr"}]: dispatch 2024-04-27T03:54:23.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:22 smithi046 ceph-mon[26899]: Manager daemon smithi046.qrfdvr is now available 2024-04-27T03:54:23.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:22 smithi046 ceph-mon[26899]: from='mgr.14100 172.21.15.46:0/2302581870' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.qrfdvr/mirror_snapshot_schedule"}]: dispatch 2024-04-27T03:54:23.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:22 smithi046 ceph-mon[26899]: from='mgr.14100 172.21.15.46:0/2302581870' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:23.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:22 smithi046 ceph-mon[26899]: from='mgr.14100 172.21.15.46:0/2302581870' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.qrfdvr/trash_purge_schedule"}]: dispatch 2024-04-27T03:54:23.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:22 smithi046 ceph-mon[26899]: from='mgr.14100 172.21.15.46:0/2302581870' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:23.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:22 smithi046 ceph-mon[26899]: from='mgr.14100 172.21.15.46:0/2302581870' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:24.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:24 smithi046 ceph-mon[26899]: mgrmap e3: smithi046.qrfdvr(active, since 1.00717s) 2024-04-27T03:54:25.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:25 smithi046 ceph-mon[26899]: mgrmap e4: smithi046.qrfdvr(active, since 2s) 2024-04-27T03:54:27.927 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-27T03:54:27.927 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-04-27T03:54:27.927 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "b1d3a2d0-0449-11ef-bc93-c7b262605968", 2024-04-27T03:54:27.927 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2024-04-27T03:54:27.927 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-27T03:54:27.927 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-27T03:54:27.927 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-27T03:54:27.927 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:27.928 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-27T03:54:27.928 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-27T03:54:27.928 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2024-04-27T03:54:27.928 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-27T03:54:27.928 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-27T03:54:27.928 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "smithi046" 2024-04-27T03:54:27.928 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-27T03:54:27.928 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-27T03:54:27.928 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-27T03:54:27.928 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T03:54:27.928 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-27T03:54:27.928 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-27T03:54:27.928 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:27.929 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-27T03:54:27.929 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T03:54:27.929 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-27T03:54:27.929 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-27T03:54:27.929 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-27T03:54:27.937 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-27T03:54:27.937 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-27T03:54:27.937 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-27T03:54:27.937 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:27.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-27T03:54:27.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-27T03:54:27.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-27T03:54:27.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-27T03:54:27.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-27T03:54:27.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-27T03:54:27.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-27T03:54:27.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-27T03:54:27.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-27T03:54:27.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:27.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-27T03:54:27.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T03:54:27.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-27T03:54:27.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-27T03:54:27.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:27.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-27T03:54:27.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-27T03:54:27.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-27T03:54:27.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-27T03:54:27.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-27T03:54:27.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-27T03:54:27.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2024-04-27T03:54:27.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-27T03:54:27.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T03:54:27.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:27.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-27T03:54:27.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T03:54:27.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2024-04-27T03:54:08.503101+0000", 2024-04-27T03:54:27.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T03:54:27.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:27.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-27T03:54:27.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-04-27T03:54:27.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.550+0000 7fe970cee700 1 Processor -- start 2024-04-27T03:54:27.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.550+0000 7fe970cee700 1 -- start start 2024-04-27T03:54:27.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.551+0000 7fe970cee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe96c1048d0 0x7fe96c104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:27.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.551+0000 7fe970cee700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c1052c0 con 0x7fe96c1048d0 2024-04-27T03:54:27.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.551+0000 7fe96a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe96c1048d0 0x7fe96c104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:27.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.551+0000 7fe96a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe96c1048d0 0x7fe96c104cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41872/0 (socket says 172.21.15.46:41872) 2024-04-27T03:54:27.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.551+0000 7fe96a59c700 1 -- 172.21.15.46:0/2445985861 learned_addr learned my addr 172.21.15.46:0/2445985861 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:27.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.551+0000 7fe96a59c700 1 -- 172.21.15.46:0/2445985861 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe96c105400 con 0x7fe96c1048d0 2024-04-27T03:54:27.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.551+0000 7fe96a59c700 1 --2- 172.21.15.46:0/2445985861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe96c1048d0 0x7fe96c104cf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe95c0098a0 tx=0x7fe95c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ff51e74eb72786 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:27.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.552+0000 7fe96959a700 1 -- 172.21.15.46:0/2445985861 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe95c004180 con 0x7fe96c1048d0 2024-04-27T03:54:27.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.552+0000 7fe96959a700 1 -- 172.21.15.46:0/2445985861 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe95c0042e0 con 0x7fe96c1048d0 2024-04-27T03:54:27.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.552+0000 7fe96959a700 1 -- 172.21.15.46:0/2445985861 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe95c011a70 con 0x7fe96c1048d0 2024-04-27T03:54:27.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.552+0000 7fe970cee700 1 -- 172.21.15.46:0/2445985861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe96c1048d0 msgr2=0x7fe96c104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:27.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.552+0000 7fe970cee700 1 --2- 172.21.15.46:0/2445985861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe96c1048d0 0x7fe96c104cf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe95c0098a0 tx=0x7fe95c009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:27.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.552+0000 7fe970cee700 1 -- 172.21.15.46:0/2445985861 shutdown_connections 2024-04-27T03:54:27.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.552+0000 7fe970cee700 1 --2- 172.21.15.46:0/2445985861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe96c1048d0 0x7fe96c104cf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:27.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.553+0000 7fe970cee700 1 -- 172.21.15.46:0/2445985861 >> 172.21.15.46:0/2445985861 conn(0x7fe96c1001f0 msgr2=0x7fe96c102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:27.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.553+0000 7fe970cee700 1 -- 172.21.15.46:0/2445985861 shutdown_connections 2024-04-27T03:54:27.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.553+0000 7fe970cee700 1 -- 172.21.15.46:0/2445985861 wait complete. 2024-04-27T03:54:27.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.553+0000 7fe970cee700 1 Processor -- start 2024-04-27T03:54:27.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.553+0000 7fe970cee700 1 -- start start 2024-04-27T03:54:27.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.554+0000 7fe970cee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe96c1048d0 0x7fe96c19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:27.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.554+0000 7fe970cee700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c19a830 con 0x7fe96c1048d0 2024-04-27T03:54:27.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.554+0000 7fe96a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe96c1048d0 0x7fe96c19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:27.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.554+0000 7fe96a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe96c1048d0 0x7fe96c19a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41886/0 (socket says 172.21.15.46:41886) 2024-04-27T03:54:27.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.554+0000 7fe96a59c700 1 -- 172.21.15.46:0/587856933 learned_addr learned my addr 172.21.15.46:0/587856933 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:27.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.554+0000 7fe96a59c700 1 -- 172.21.15.46:0/587856933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95c009550 con 0x7fe96c1048d0 2024-04-27T03:54:27.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.554+0000 7fe96a59c700 1 --2- 172.21.15.46:0/587856933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe96c1048d0 0x7fe96c19a2f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe95c011710 tx=0x7fe95c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:27.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.554+0000 7fe95b7fe700 1 -- 172.21.15.46:0/587856933 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe95c003b10 con 0x7fe96c1048d0 2024-04-27T03:54:27.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.554+0000 7fe95b7fe700 1 -- 172.21.15.46:0/587856933 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe95c01b8c0 con 0x7fe96c1048d0 2024-04-27T03:54:27.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.554+0000 7fe970cee700 1 -- 172.21.15.46:0/587856933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96c19aa90 con 0x7fe96c1048d0 2024-04-27T03:54:27.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.555+0000 7fe95b7fe700 1 -- 172.21.15.46:0/587856933 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe95c019770 con 0x7fe96c1048d0 2024-04-27T03:54:27.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.555+0000 7fe970cee700 1 -- 172.21.15.46:0/587856933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96c19d760 con 0x7fe96c1048d0 2024-04-27T03:54:27.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.555+0000 7fe95b7fe700 1 -- 172.21.15.46:0/587856933 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fe95c019ca0 con 0x7fe96c1048d0 2024-04-27T03:54:27.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.555+0000 7fe95b7fe700 1 -- 172.21.15.46:0/587856933 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe95c01a830 con 0x7fe96c1048d0 2024-04-27T03:54:27.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.556+0000 7fe970cee700 1 -- 172.21.15.46:0/587856933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe96c062510 con 0x7fe96c1048d0 2024-04-27T03:54:27.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.558+0000 7fe95b7fe700 1 -- 172.21.15.46:0/587856933 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fe95c029060 con 0x7fe96c1048d0 2024-04-27T03:54:27.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.632+0000 7fe970cee700 1 -- 172.21.15.46:0/587856933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe96c108520 con 0x7fe96c1048d0 2024-04-27T03:54:27.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.632+0000 7fe95b7fe700 1 -- 172.21.15.46:0/587856933 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fe95c01f070 con 0x7fe96c1048d0 2024-04-27T03:54:27.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.633+0000 7fe970cee700 1 -- 172.21.15.46:0/587856933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe96c1048d0 msgr2=0x7fe96c19a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:27.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.633+0000 7fe970cee700 1 --2- 172.21.15.46:0/587856933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe96c1048d0 0x7fe96c19a2f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe95c011710 tx=0x7fe95c0047c0 comp rx=0 tx=0).stop 2024-04-27T03:54:27.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.633+0000 7fe970cee700 1 -- 172.21.15.46:0/587856933 shutdown_connections 2024-04-27T03:54:27.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.633+0000 7fe970cee700 1 --2- 172.21.15.46:0/587856933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe96c1048d0 0x7fe96c19a2f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:27.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.633+0000 7fe970cee700 1 -- 172.21.15.46:0/587856933 >> 172.21.15.46:0/587856933 conn(0x7fe96c1001f0 msgr2=0x7fe96c100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:27.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.633+0000 7fe970cee700 1 -- 172.21.15.46:0/587856933 shutdown_connections 2024-04-27T03:54:27.944 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:21.633+0000 7fe970cee700 1 -- 172.21.15.46:0/587856933 wait complete. 2024-04-27T03:54:27.944 INFO:teuthology.orchestra.run.smithi046.stdout:mgr not available, waiting (2/15)... 2024-04-27T03:54:30.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:30 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/1976358990' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T03:54:31.213 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-27T03:54:31.213 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-04-27T03:54:31.213 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "b1d3a2d0-0449-11ef-bc93-c7b262605968", 2024-04-27T03:54:31.213 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2024-04-27T03:54:31.213 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-27T03:54:31.213 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-27T03:54:31.213 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-27T03:54:31.213 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:31.213 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-27T03:54:31.213 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-27T03:54:31.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2024-04-27T03:54:31.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-27T03:54:31.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-27T03:54:31.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "smithi046" 2024-04-27T03:54:31.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-27T03:54:31.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 17, 2024-04-27T03:54:31.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-27T03:54:31.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T03:54:31.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-27T03:54:31.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-27T03:54:31.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:31.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-27T03:54:31.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T03:54:31.215 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-27T03:54:31.215 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-27T03:54:31.215 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-27T03:54:31.215 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-27T03:54:31.215 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-27T03:54:31.215 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-27T03:54:31.215 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:31.215 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-27T03:54:31.215 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-27T03:54:31.215 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-27T03:54:31.215 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-27T03:54:31.215 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-27T03:54:31.215 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-27T03:54:31.215 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-27T03:54:31.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-27T03:54:31.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-27T03:54:31.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:31.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-27T03:54:31.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T03:54:31.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-27T03:54:31.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-27T03:54:31.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:31.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-27T03:54:31.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-27T03:54:31.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-27T03:54:31.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-27T03:54:31.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-27T03:54:31.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-27T03:54:31.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2024-04-27T03:54:31.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-27T03:54:31.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T03:54:31.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:31.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-27T03:54:31.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T03:54:31.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2024-04-27T03:54:08.503101+0000", 2024-04-27T03:54:31.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T03:54:31.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-27T03:54:31.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-27T03:54:31.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-04-27T03:54:31.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.368+0000 7fbcea48a700 1 Processor -- start 2024-04-27T03:54:31.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.368+0000 7fbcea48a700 1 -- start start 2024-04-27T03:54:31.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.368+0000 7fbcea48a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce407bec0 0x7fbce407a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:31.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.368+0000 7fbcea48a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce407a8f0 con 0x7fbce407bec0 2024-04-27T03:54:31.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.368+0000 7fbce3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce407bec0 0x7fbce407a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:31.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.368+0000 7fbce3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce407bec0 0x7fbce407a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41958/0 (socket says 172.21.15.46:41958) 2024-04-27T03:54:31.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.368+0000 7fbce3fff700 1 -- 172.21.15.46:0/1029144311 learned_addr learned my addr 172.21.15.46:0/1029144311 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:31.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.368+0000 7fbce3fff700 1 -- 172.21.15.46:0/1029144311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce407aa30 con 0x7fbce407bec0 2024-04-27T03:54:31.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.369+0000 7fbce3fff700 1 --2- 172.21.15.46:0/1029144311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce407bec0 0x7fbce407a320 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbcd40098a0 tx=0x7fbcd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e8582aa24b4fe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:31.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.369+0000 7fbce2ffd700 1 -- 172.21.15.46:0/1029144311 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcd4004180 con 0x7fbce407bec0 2024-04-27T03:54:31.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.369+0000 7fbce2ffd700 1 -- 172.21.15.46:0/1029144311 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fbcd40042e0 con 0x7fbce407bec0 2024-04-27T03:54:31.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.369+0000 7fbce2ffd700 1 -- 172.21.15.46:0/1029144311 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcd4011a70 con 0x7fbce407bec0 2024-04-27T03:54:31.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.369+0000 7fbcea48a700 1 -- 172.21.15.46:0/1029144311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce407bec0 msgr2=0x7fbce407a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:31.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.369+0000 7fbcea48a700 1 --2- 172.21.15.46:0/1029144311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce407bec0 0x7fbce407a320 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbcd40098a0 tx=0x7fbcd4009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:31.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.369+0000 7fbcea48a700 1 -- 172.21.15.46:0/1029144311 shutdown_connections 2024-04-27T03:54:31.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.370+0000 7fbcea48a700 1 --2- 172.21.15.46:0/1029144311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce407bec0 0x7fbce407a320 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:31.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.370+0000 7fbcea48a700 1 -- 172.21.15.46:0/1029144311 >> 172.21.15.46:0/1029144311 conn(0x7fbce4101be0 msgr2=0x7fbce4104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:31.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.370+0000 7fbcea48a700 1 -- 172.21.15.46:0/1029144311 shutdown_connections 2024-04-27T03:54:31.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.370+0000 7fbcea48a700 1 -- 172.21.15.46:0/1029144311 wait complete. 2024-04-27T03:54:31.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.370+0000 7fbcea48a700 1 Processor -- start 2024-04-27T03:54:31.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.370+0000 7fbcea48a700 1 -- start start 2024-04-27T03:54:31.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.371+0000 7fbcea48a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce407bec0 0x7fbce419c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:31.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.371+0000 7fbcea48a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce419ca30 con 0x7fbce407bec0 2024-04-27T03:54:31.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.371+0000 7fbce3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce407bec0 0x7fbce419c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:31.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.371+0000 7fbce3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce407bec0 0x7fbce419c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41962/0 (socket says 172.21.15.46:41962) 2024-04-27T03:54:31.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.371+0000 7fbce3fff700 1 -- 172.21.15.46:0/1976358990 learned_addr learned my addr 172.21.15.46:0/1976358990 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:31.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.371+0000 7fbce3fff700 1 -- 172.21.15.46:0/1976358990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd4009550 con 0x7fbce407bec0 2024-04-27T03:54:31.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.371+0000 7fbce3fff700 1 --2- 172.21.15.46:0/1976358990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce407bec0 0x7fbce419c4f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4011710 tx=0x7fbcd40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:31.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.371+0000 7fbce17fa700 1 -- 172.21.15.46:0/1976358990 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcd4003b10 con 0x7fbce407bec0 2024-04-27T03:54:31.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.371+0000 7fbce17fa700 1 -- 172.21.15.46:0/1976358990 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fbcd401b8c0 con 0x7fbce407bec0 2024-04-27T03:54:31.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.371+0000 7fbcea48a700 1 -- 172.21.15.46:0/1976358990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce419cc30 con 0x7fbce407bec0 2024-04-27T03:54:31.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.371+0000 7fbce17fa700 1 -- 172.21.15.46:0/1976358990 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcd4019710 con 0x7fbce407bec0 2024-04-27T03:54:31.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.371+0000 7fbcea48a700 1 -- 172.21.15.46:0/1976358990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce419d0d0 con 0x7fbce407bec0 2024-04-27T03:54:31.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.372+0000 7fbce17fa700 1 -- 172.21.15.46:0/1976358990 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 45545+0+0 (secure 0 0 0) 0x7fbcd4019930 con 0x7fbce407bec0 2024-04-27T03:54:31.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.372+0000 7fbce17fa700 1 --2- 172.21.15.46:0/1976358990 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fbcd0038690 0x7fbcd003ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:31.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.372+0000 7fbce17fa700 1 -- 172.21.15.46:0/1976358990 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbcd404cfa0 con 0x7fbce407bec0 2024-04-27T03:54:31.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.373+0000 7fbcea48a700 1 -- 172.21.15.46:0/1976358990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce410c920 con 0x7fbce407bec0 2024-04-27T03:54:31.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.373+0000 7fbce37fe700 1 --2- 172.21.15.46:0/1976358990 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fbcd0038690 0x7fbcd003ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:31.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.373+0000 7fbce37fe700 1 --2- 172.21.15.46:0/1976358990 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fbcd0038690 0x7fbcd003ab50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbccc006fd0 tx=0x7fbccc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:31.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.378+0000 7fbce17fa700 1 -- 172.21.15.46:0/1976358990 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fbcd4018d90 con 0x7fbce407bec0 2024-04-27T03:54:31.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.662+0000 7fbcea48a700 1 -- 172.21.15.46:0/1976358990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbce419fc60 con 0x7fbce407bec0 2024-04-27T03:54:31.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.662+0000 7fbce17fa700 1 -- 172.21.15.46:0/1976358990 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fbcd4018350 con 0x7fbce407bec0 2024-04-27T03:54:31.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.664+0000 7fbcea48a700 1 -- 172.21.15.46:0/1976358990 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fbcd0038690 msgr2=0x7fbcd003ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:31.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.664+0000 7fbcea48a700 1 --2- 172.21.15.46:0/1976358990 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fbcd0038690 0x7fbcd003ab50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbccc006fd0 tx=0x7fbccc006e40 comp rx=0 tx=0).stop 2024-04-27T03:54:31.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.665+0000 7fbcea48a700 1 -- 172.21.15.46:0/1976358990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce407bec0 msgr2=0x7fbce419c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:31.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.665+0000 7fbcea48a700 1 --2- 172.21.15.46:0/1976358990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce407bec0 0x7fbce419c4f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4011710 tx=0x7fbcd40047c0 comp rx=0 tx=0).stop 2024-04-27T03:54:31.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.665+0000 7fbcea48a700 1 -- 172.21.15.46:0/1976358990 shutdown_connections 2024-04-27T03:54:31.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.665+0000 7fbcea48a700 1 --2- 172.21.15.46:0/1976358990 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fbcd0038690 0x7fbcd003ab50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:31.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.665+0000 7fbcea48a700 1 --2- 172.21.15.46:0/1976358990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce407bec0 0x7fbce419c4f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:31.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.665+0000 7fbcea48a700 1 -- 172.21.15.46:0/1976358990 >> 172.21.15.46:0/1976358990 conn(0x7fbce4101be0 msgr2=0x7fbce41026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:31.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.665+0000 7fbcea48a700 1 -- 172.21.15.46:0/1976358990 shutdown_connections 2024-04-27T03:54:31.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:30.665+0000 7fbcea48a700 1 -- 172.21.15.46:0/1976358990 wait complete. 2024-04-27T03:54:31.222 INFO:teuthology.orchestra.run.smithi046.stdout:mgr is available 2024-04-27T03:54:32.579 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:32 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/1617029955' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-27T03:54:32.857 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-27T03:54:32.857 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [global] 2024-04-27T03:54:32.857 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout fsid = b1d3a2d0-0449-11ef-bc93-c7b262605968 2024-04-27T03:54:32.857 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-27T03:54:32.857 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-27T03:54:32.857 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-27T03:54:32.857 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-27T03:54:32.857 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-27T03:54:32.857 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-27T03:54:32.857 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-27T03:54:32.858 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-27T03:54:32.858 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [osd] 2024-04-27T03:54:32.858 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-27T03:54:32.858 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-27T03:54:32.858 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-27T03:54:32.858 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-27T03:54:32.858 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-27T03:54:32.858 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.061+0000 7f1878b1a700 1 Processor -- start 2024-04-27T03:54:32.858 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.061+0000 7f1878b1a700 1 -- start start 2024-04-27T03:54:32.858 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.062+0000 7f1878b1a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18741065c0 0x7f18741069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:32.858 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.062+0000 7f1878b1a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1874106fb0 con 0x7f18741065c0 2024-04-27T03:54:32.859 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.062+0000 7f187259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18741065c0 0x7f18741069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:32.859 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.062+0000 7f187259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18741065c0 0x7f18741069e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58408/0 (socket says 172.21.15.46:58408) 2024-04-27T03:54:32.859 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.062+0000 7f187259c700 1 -- 172.21.15.46:0/1537760216 learned_addr learned my addr 172.21.15.46:0/1537760216 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:32.859 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.062+0000 7f187259c700 1 -- 172.21.15.46:0/1537760216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18741077c0 con 0x7f18741065c0 2024-04-27T03:54:32.859 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.063+0000 7f187259c700 1 --2- 172.21.15.46:0/1537760216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18741065c0 0x7f18741069e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f18680098a0 tx=0x7f1868009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b10a5d377acdfb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:32.859 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.063+0000 7f187159a700 1 -- 172.21.15.46:0/1537760216 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1868004180 con 0x7f18741065c0 2024-04-27T03:54:32.859 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.063+0000 7f187159a700 1 -- 172.21.15.46:0/1537760216 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f18680042e0 con 0x7f18741065c0 2024-04-27T03:54:32.859 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.063+0000 7f187159a700 1 -- 172.21.15.46:0/1537760216 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1868011a70 con 0x7f18741065c0 2024-04-27T03:54:32.860 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.063+0000 7f1878b1a700 1 -- 172.21.15.46:0/1537760216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18741065c0 msgr2=0x7f18741069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:32.860 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.063+0000 7f1878b1a700 1 --2- 172.21.15.46:0/1537760216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18741065c0 0x7f18741069e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f18680098a0 tx=0x7f1868009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:32.860 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.063+0000 7f1878b1a700 1 -- 172.21.15.46:0/1537760216 shutdown_connections 2024-04-27T03:54:32.860 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.063+0000 7f1878b1a700 1 --2- 172.21.15.46:0/1537760216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18741065c0 0x7f18741069e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:32.860 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.063+0000 7f1878b1a700 1 -- 172.21.15.46:0/1537760216 >> 172.21.15.46:0/1537760216 conn(0x7f1874101be0 msgr2=0x7f1874104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:32.861 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.064+0000 7f1878b1a700 1 -- 172.21.15.46:0/1537760216 shutdown_connections 2024-04-27T03:54:32.861 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.064+0000 7f1878b1a700 1 -- 172.21.15.46:0/1537760216 wait complete. 2024-04-27T03:54:32.861 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.064+0000 7f1878b1a700 1 Processor -- start 2024-04-27T03:54:32.861 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.064+0000 7f1878b1a700 1 -- start start 2024-04-27T03:54:32.861 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.065+0000 7f1878b1a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18741065c0 0x7f187419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:32.861 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.065+0000 7f1878b1a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187419a830 con 0x7f18741065c0 2024-04-27T03:54:32.861 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.065+0000 7f187259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18741065c0 0x7f187419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:32.861 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.065+0000 7f187259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18741065c0 0x7f187419a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58414/0 (socket says 172.21.15.46:58414) 2024-04-27T03:54:32.861 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.065+0000 7f187259c700 1 -- 172.21.15.46:0/1617029955 learned_addr learned my addr 172.21.15.46:0/1617029955 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:32.861 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.065+0000 7f187259c700 1 -- 172.21.15.46:0/1617029955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1868009550 con 0x7f18741065c0 2024-04-27T03:54:32.862 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.065+0000 7f187259c700 1 --2- 172.21.15.46:0/1617029955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18741065c0 0x7f187419a2f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1868011710 tx=0x7f18680047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:32.862 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.065+0000 7f18637fe700 1 -- 172.21.15.46:0/1617029955 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1868003b10 con 0x7f18741065c0 2024-04-27T03:54:32.862 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.065+0000 7f18637fe700 1 -- 172.21.15.46:0/1617029955 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f186801b8c0 con 0x7f18741065c0 2024-04-27T03:54:32.862 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.065+0000 7f1878b1a700 1 -- 172.21.15.46:0/1617029955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187419aa30 con 0x7f18741065c0 2024-04-27T03:54:32.862 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.065+0000 7f18637fe700 1 -- 172.21.15.46:0/1617029955 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1868019770 con 0x7f18741065c0 2024-04-27T03:54:32.862 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.065+0000 7f1878b1a700 1 -- 172.21.15.46:0/1617029955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f187419aed0 con 0x7f18741065c0 2024-04-27T03:54:32.862 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.066+0000 7f18637fe700 1 -- 172.21.15.46:0/1617029955 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 45545+0+0 (secure 0 0 0) 0x7f18680199f0 con 0x7f18741065c0 2024-04-27T03:54:32.862 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.066+0000 7f1878b1a700 1 -- 172.21.15.46:0/1617029955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f187410a9f0 con 0x7f18741065c0 2024-04-27T03:54:32.862 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.066+0000 7f18637fe700 1 --2- 172.21.15.46:0/1617029955 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7f185c0386d0 0x7f185c03ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:32.862 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.067+0000 7f18637fe700 1 -- 172.21.15.46:0/1617029955 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f186804c8f0 con 0x7f18741065c0 2024-04-27T03:54:32.862 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.067+0000 7f1871d9b700 1 --2- 172.21.15.46:0/1617029955 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7f185c0386d0 0x7f185c03ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:32.862 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.067+0000 7f1871d9b700 1 --2- 172.21.15.46:0/1617029955 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7f185c0386d0 0x7f185c03ab90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1864006fd0 tx=0x7f1864006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:32.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.072+0000 7f18637fe700 1 -- 172.21.15.46:0/1617029955 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f1868018db0 con 0x7f18741065c0 2024-04-27T03:54:32.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.271+0000 7f1878b1a700 1 -- 172.21.15.46:0/1617029955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f187419d760 con 0x7f18741065c0 2024-04-27T03:54:32.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.272+0000 7f18637fe700 1 -- 172.21.15.46:0/1617029955 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f186801f070 con 0x7f18741065c0 2024-04-27T03:54:32.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.274+0000 7f1878b1a700 1 -- 172.21.15.46:0/1617029955 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7f185c0386d0 msgr2=0x7f185c03ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:32.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.274+0000 7f1878b1a700 1 --2- 172.21.15.46:0/1617029955 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7f185c0386d0 0x7f185c03ab90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1864006fd0 tx=0x7f1864006e40 comp rx=0 tx=0).stop 2024-04-27T03:54:32.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.274+0000 7f1878b1a700 1 -- 172.21.15.46:0/1617029955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18741065c0 msgr2=0x7f187419a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:32.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.274+0000 7f1878b1a700 1 --2- 172.21.15.46:0/1617029955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18741065c0 0x7f187419a2f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1868011710 tx=0x7f18680047c0 comp rx=0 tx=0).stop 2024-04-27T03:54:32.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.274+0000 7f1878b1a700 1 -- 172.21.15.46:0/1617029955 shutdown_connections 2024-04-27T03:54:32.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.274+0000 7f1878b1a700 1 --2- 172.21.15.46:0/1617029955 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7f185c0386d0 0x7f185c03ab90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:32.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.274+0000 7f1878b1a700 1 --2- 172.21.15.46:0/1617029955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18741065c0 0x7f187419a2f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:32.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.274+0000 7f1878b1a700 1 -- 172.21.15.46:0/1617029955 >> 172.21.15.46:0/1617029955 conn(0x7f1874101be0 msgr2=0x7f18741026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:32.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.274+0000 7f1878b1a700 1 -- 172.21.15.46:0/1617029955 shutdown_connections 2024-04-27T03:54:32.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:32.274+0000 7f1878b1a700 1 -- 172.21.15.46:0/1617029955 wait complete. 2024-04-27T03:54:32.864 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling cephadm module... 2024-04-27T03:54:33.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:33 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/95783006' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-27T03:54:34.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:34 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/95783006' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-27T03:54:34.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:34 smithi046 ceph-mon[26899]: mgrmap e5: smithi046.qrfdvr(active, since 11s) 2024-04-27T03:54:34.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.241+0000 7f041a4ee700 1 Processor -- start 2024-04-27T03:54:34.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.241+0000 7f041a4ee700 1 -- start start 2024-04-27T03:54:34.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.242+0000 7f041a4ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0414104e00 0x7f0414105220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:34.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.242+0000 7f041a4ee700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04140747b0 con 0x7f0414104e00 2024-04-27T03:54:34.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.242+0000 7f0413fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0414104e00 0x7f0414105220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:34.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.242+0000 7f0413fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0414104e00 0x7f0414105220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58416/0 (socket says 172.21.15.46:58416) 2024-04-27T03:54:34.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.242+0000 7f0413fff700 1 -- 172.21.15.46:0/2656593562 learned_addr learned my addr 172.21.15.46:0/2656593562 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:34.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.242+0000 7f0413fff700 1 -- 172.21.15.46:0/2656593562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0414105760 con 0x7f0414104e00 2024-04-27T03:54:34.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.242+0000 7f0413fff700 1 --2- 172.21.15.46:0/2656593562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0414104e00 0x7f0414105220 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f04040098a0 tx=0x7f0404009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0bfff82976a461 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:34.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.243+0000 7f0412ffd700 1 -- 172.21.15.46:0/2656593562 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0404004180 con 0x7f0414104e00 2024-04-27T03:54:34.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.243+0000 7f0412ffd700 1 -- 172.21.15.46:0/2656593562 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f04040042e0 con 0x7f0414104e00 2024-04-27T03:54:34.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.243+0000 7f0412ffd700 1 -- 172.21.15.46:0/2656593562 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0404011a70 con 0x7f0414104e00 2024-04-27T03:54:34.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.243+0000 7f041a4ee700 1 -- 172.21.15.46:0/2656593562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0414104e00 msgr2=0x7f0414105220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:34.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.243+0000 7f041a4ee700 1 --2- 172.21.15.46:0/2656593562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0414104e00 0x7f0414105220 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f04040098a0 tx=0x7f0404009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:34.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.243+0000 7f041a4ee700 1 -- 172.21.15.46:0/2656593562 shutdown_connections 2024-04-27T03:54:34.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.243+0000 7f041a4ee700 1 --2- 172.21.15.46:0/2656593562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0414104e00 0x7f0414105220 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:34.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.243+0000 7f041a4ee700 1 -- 172.21.15.46:0/2656593562 >> 172.21.15.46:0/2656593562 conn(0x7f0414100a60 msgr2=0x7f0414102e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:34.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.243+0000 7f041a4ee700 1 -- 172.21.15.46:0/2656593562 shutdown_connections 2024-04-27T03:54:34.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.243+0000 7f041a4ee700 1 -- 172.21.15.46:0/2656593562 wait complete. 2024-04-27T03:54:34.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.244+0000 7f041a4ee700 1 Processor -- start 2024-04-27T03:54:34.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.244+0000 7f041a4ee700 1 -- start start 2024-04-27T03:54:34.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.244+0000 7f041a4ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0414104e00 0x7f041419c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:34.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.244+0000 7f041a4ee700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041419ca20 con 0x7f0414104e00 2024-04-27T03:54:34.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.244+0000 7f0413fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0414104e00 0x7f041419c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:34.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.244+0000 7f0413fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0414104e00 0x7f041419c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58426/0 (socket says 172.21.15.46:58426) 2024-04-27T03:54:34.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.244+0000 7f0413fff700 1 -- 172.21.15.46:0/95783006 learned_addr learned my addr 172.21.15.46:0/95783006 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:34.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.245+0000 7f0413fff700 1 -- 172.21.15.46:0/95783006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0404009550 con 0x7f0414104e00 2024-04-27T03:54:34.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.245+0000 7f0413fff700 1 --2- 172.21.15.46:0/95783006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0414104e00 0x7f041419c4e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0404011710 tx=0x7f04040047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:34.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.245+0000 7f04117fa700 1 -- 172.21.15.46:0/95783006 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0404003b10 con 0x7f0414104e00 2024-04-27T03:54:34.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.245+0000 7f04117fa700 1 -- 172.21.15.46:0/95783006 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f040401b8c0 con 0x7f0414104e00 2024-04-27T03:54:34.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.245+0000 7f041a4ee700 1 -- 172.21.15.46:0/95783006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f041419cc20 con 0x7f0414104e00 2024-04-27T03:54:34.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.245+0000 7f04117fa700 1 -- 172.21.15.46:0/95783006 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04040197e0 con 0x7f0414104e00 2024-04-27T03:54:34.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.245+0000 7f041a4ee700 1 -- 172.21.15.46:0/95783006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f041419d0c0 con 0x7f0414104e00 2024-04-27T03:54:34.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.246+0000 7f04117fa700 1 -- 172.21.15.46:0/95783006 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 45545+0+0 (secure 0 0 0) 0x7f040401b440 con 0x7f0414104e00 2024-04-27T03:54:34.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.246+0000 7f041a4ee700 1 -- 172.21.15.46:0/95783006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0414196170 con 0x7f0414104e00 2024-04-27T03:54:34.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.246+0000 7f04117fa700 1 --2- 172.21.15.46:0/95783006 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7f0400038720 0x7f040003abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:34.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.246+0000 7f04117fa700 1 -- 172.21.15.46:0/95783006 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f040404c8c0 con 0x7f0414104e00 2024-04-27T03:54:34.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.246+0000 7f04137fe700 1 --2- 172.21.15.46:0/95783006 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7f0400038720 0x7f040003abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:34.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.247+0000 7f04137fe700 1 --2- 172.21.15.46:0/95783006 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7f0400038720 0x7f040003abe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f03fc006fd0 tx=0x7f03fc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:34.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.251+0000 7f04117fa700 1 -- 172.21.15.46:0/95783006 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f04040199d0 con 0x7f0414104e00 2024-04-27T03:54:34.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.514+0000 7f041a4ee700 1 -- 172.21.15.46:0/95783006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f0414106580 con 0x7f0414104e00 2024-04-27T03:54:34.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.566+0000 7f04117fa700 1 -- 172.21.15.46:0/95783006 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 5) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f0404019d50 con 0x7f0414104e00 2024-04-27T03:54:34.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.568+0000 7f04117fa700 1 -- 172.21.15.46:0/95783006 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f0404016dc0 con 0x7f0414104e00 2024-04-27T03:54:34.779 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.570+0000 7f041a4ee700 1 -- 172.21.15.46:0/95783006 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7f0400038720 msgr2=0x7f040003abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:34.779 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.570+0000 7f041a4ee700 1 --2- 172.21.15.46:0/95783006 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7f0400038720 0x7f040003abe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f03fc006fd0 tx=0x7f03fc006e40 comp rx=0 tx=0).stop 2024-04-27T03:54:34.779 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.570+0000 7f041a4ee700 1 -- 172.21.15.46:0/95783006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0414104e00 msgr2=0x7f041419c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:34.779 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.570+0000 7f041a4ee700 1 --2- 172.21.15.46:0/95783006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0414104e00 0x7f041419c4e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0404011710 tx=0x7f04040047c0 comp rx=0 tx=0).stop 2024-04-27T03:54:34.779 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.571+0000 7f041a4ee700 1 -- 172.21.15.46:0/95783006 shutdown_connections 2024-04-27T03:54:34.779 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.571+0000 7f041a4ee700 1 --2- 172.21.15.46:0/95783006 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7f0400038720 0x7f040003abe0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:34.779 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.571+0000 7f041a4ee700 1 --2- 172.21.15.46:0/95783006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0414104e00 0x7f041419c4e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:34.779 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.571+0000 7f041a4ee700 1 -- 172.21.15.46:0/95783006 >> 172.21.15.46:0/95783006 conn(0x7f0414100a60 msgr2=0x7f04141932b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:34.779 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.571+0000 7f041a4ee700 1 -- 172.21.15.46:0/95783006 shutdown_connections 2024-04-27T03:54:34.779 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:33.571+0000 7f041a4ee700 1 -- 172.21.15.46:0/95783006 wait complete. 2024-04-27T03:54:35.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:35 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/654721072' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-27T03:54:36.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-04-27T03:54:36.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-27T03:54:36.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-27T03:54:36.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "active_name": "smithi046.qrfdvr", 2024-04-27T03:54:36.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-27T03:54:36.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-04-27T03:54:36.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.185+0000 7fe319aaf700 1 Processor -- start 2024-04-27T03:54:36.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.185+0000 7fe319aaf700 1 -- start start 2024-04-27T03:54:36.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.185+0000 7fe319aaf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe314104e10 0x7fe314105230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:36.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.185+0000 7fe319aaf700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3140747b0 con 0x7fe314104e10 2024-04-27T03:54:36.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.186+0000 7fe3137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe314104e10 0x7fe314105230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:36.094 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.186+0000 7fe3137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe314104e10 0x7fe314105230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58446/0 (socket says 172.21.15.46:58446) 2024-04-27T03:54:36.094 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.186+0000 7fe3137fe700 1 -- 172.21.15.46:0/1114292528 learned_addr learned my addr 172.21.15.46:0/1114292528 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:36.094 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.186+0000 7fe3137fe700 1 -- 172.21.15.46:0/1114292528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe314105770 con 0x7fe314104e10 2024-04-27T03:54:36.094 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.186+0000 7fe3137fe700 1 --2- 172.21.15.46:0/1114292528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe314104e10 0x7fe314105230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe3040098a0 tx=0x7fe304009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d112cf590f9f51d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:36.094 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.186+0000 7fe3127fc700 1 -- 172.21.15.46:0/1114292528 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe304004180 con 0x7fe314104e10 2024-04-27T03:54:36.094 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.186+0000 7fe3127fc700 1 -- 172.21.15.46:0/1114292528 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe3040042e0 con 0x7fe314104e10 2024-04-27T03:54:36.094 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.186+0000 7fe3127fc700 1 -- 172.21.15.46:0/1114292528 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe304011a70 con 0x7fe314104e10 2024-04-27T03:54:36.094 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.187+0000 7fe319aaf700 1 -- 172.21.15.46:0/1114292528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe314104e10 msgr2=0x7fe314105230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:36.094 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.187+0000 7fe319aaf700 1 --2- 172.21.15.46:0/1114292528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe314104e10 0x7fe314105230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe3040098a0 tx=0x7fe304009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:36.094 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.187+0000 7fe319aaf700 1 -- 172.21.15.46:0/1114292528 shutdown_connections 2024-04-27T03:54:36.094 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.187+0000 7fe319aaf700 1 --2- 172.21.15.46:0/1114292528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe314104e10 0x7fe314105230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:36.094 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.187+0000 7fe319aaf700 1 -- 172.21.15.46:0/1114292528 >> 172.21.15.46:0/1114292528 conn(0x7fe314100a50 msgr2=0x7fe314102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:36.095 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.187+0000 7fe319aaf700 1 -- 172.21.15.46:0/1114292528 shutdown_connections 2024-04-27T03:54:36.095 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.187+0000 7fe319aaf700 1 -- 172.21.15.46:0/1114292528 wait complete. 2024-04-27T03:54:36.095 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.188+0000 7fe319aaf700 1 Processor -- start 2024-04-27T03:54:36.095 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.188+0000 7fe319aaf700 1 -- start start 2024-04-27T03:54:36.095 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.188+0000 7fe319aaf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe314104e10 0x7fe31419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:36.095 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.188+0000 7fe319aaf700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31419ca40 con 0x7fe314104e10 2024-04-27T03:54:36.095 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.188+0000 7fe3137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe314104e10 0x7fe31419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:36.095 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.188+0000 7fe3137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe314104e10 0x7fe31419c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58456/0 (socket says 172.21.15.46:58456) 2024-04-27T03:54:36.095 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.188+0000 7fe3137fe700 1 -- 172.21.15.46:0/654721072 learned_addr learned my addr 172.21.15.46:0/654721072 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:36.095 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.189+0000 7fe3137fe700 1 -- 172.21.15.46:0/654721072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe304009550 con 0x7fe314104e10 2024-04-27T03:54:36.095 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.189+0000 7fe3137fe700 1 --2- 172.21.15.46:0/654721072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe314104e10 0x7fe31419c500 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe304011710 tx=0x7fe3040047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:36.095 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.189+0000 7fe310ff9700 1 -- 172.21.15.46:0/654721072 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe304003b10 con 0x7fe314104e10 2024-04-27T03:54:36.095 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.189+0000 7fe310ff9700 1 -- 172.21.15.46:0/654721072 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe30401b8c0 con 0x7fe314104e10 2024-04-27T03:54:36.096 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.189+0000 7fe319aaf700 1 -- 172.21.15.46:0/654721072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31419cc40 con 0x7fe314104e10 2024-04-27T03:54:36.096 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.190+0000 7fe310ff9700 1 -- 172.21.15.46:0/654721072 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3040197e0 con 0x7fe314104e10 2024-04-27T03:54:36.096 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.190+0000 7fe319aaf700 1 -- 172.21.15.46:0/654721072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31419d0e0 con 0x7fe314104e10 2024-04-27T03:54:36.096 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.190+0000 7fe310ff9700 1 -- 172.21.15.46:0/654721072 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fe3040199f0 con 0x7fe314104e10 2024-04-27T03:54:36.096 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.190+0000 7fe310ff9700 1 --2- 172.21.15.46:0/654721072 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fe2fc0387c0 0x7fe2fc03ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:36.096 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.190+0000 7fe310ff9700 1 -- 172.21.15.46:0/654721072 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe3040521d0 con 0x7fe314104e10 2024-04-27T03:54:36.096 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.190+0000 7fe312ffd700 1 -- 172.21.15.46:0/654721072 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fe2fc0387c0 msgr2=0x7fe2fc03ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3824329935 2024-04-27T03:54:36.096 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.190+0000 7fe312ffd700 1 --2- 172.21.15.46:0/654721072 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fe2fc0387c0 0x7fe2fc03ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-27T03:54:36.096 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.191+0000 7fe319aaf700 1 -- 172.21.15.46:0/654721072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe314196180 con 0x7fe314104e10 2024-04-27T03:54:36.096 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.200+0000 7fe310ff9700 1 -- 172.21.15.46:0/654721072 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fe30401a8f0 con 0x7fe314104e10 2024-04-27T03:54:36.096 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.391+0000 7fe312ffd700 1 -- 172.21.15.46:0/654721072 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fe2fc0387c0 msgr2=0x7fe2fc03ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3824329935 2024-04-27T03:54:36.096 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.391+0000 7fe312ffd700 1 --2- 172.21.15.46:0/654721072 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fe2fc0387c0 0x7fe2fc03ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-27T03:54:36.096 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.463+0000 7fe319aaf700 1 -- 172.21.15.46:0/654721072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fe314106810 con 0x7fe314104e10 2024-04-27T03:54:36.097 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.464+0000 7fe310ff9700 1 -- 172.21.15.46:0/654721072 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fe30401e170 con 0x7fe314104e10 2024-04-27T03:54:36.097 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.466+0000 7fe319aaf700 1 -- 172.21.15.46:0/654721072 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fe2fc0387c0 msgr2=0x7fe2fc03ac80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T03:54:36.097 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.466+0000 7fe319aaf700 1 --2- 172.21.15.46:0/654721072 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fe2fc0387c0 0x7fe2fc03ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:36.097 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.466+0000 7fe319aaf700 1 -- 172.21.15.46:0/654721072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe314104e10 msgr2=0x7fe31419c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:36.097 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.466+0000 7fe319aaf700 1 --2- 172.21.15.46:0/654721072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe314104e10 0x7fe31419c500 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe304011710 tx=0x7fe3040047c0 comp rx=0 tx=0).stop 2024-04-27T03:54:36.097 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.466+0000 7fe319aaf700 1 -- 172.21.15.46:0/654721072 shutdown_connections 2024-04-27T03:54:36.097 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.466+0000 7fe319aaf700 1 --2- 172.21.15.46:0/654721072 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fe2fc0387c0 0x7fe2fc03ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:36.097 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.466+0000 7fe319aaf700 1 --2- 172.21.15.46:0/654721072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe314104e10 0x7fe31419c500 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:36.097 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.466+0000 7fe319aaf700 1 -- 172.21.15.46:0/654721072 >> 172.21.15.46:0/654721072 conn(0x7fe314100a50 msgr2=0x7fe3141932c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:36.097 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.466+0000 7fe319aaf700 1 -- 172.21.15.46:0/654721072 shutdown_connections 2024-04-27T03:54:36.097 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:35.466+0000 7fe319aaf700 1 -- 172.21.15.46:0/654721072 wait complete. 2024-04-27T03:54:36.097 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for the mgr to restart... 2024-04-27T03:54:36.097 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr epoch 5... 2024-04-27T03:54:41.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:40 smithi046 ceph-mon[26899]: Active manager daemon smithi046.qrfdvr restarted 2024-04-27T03:54:41.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:40 smithi046 ceph-mon[26899]: Activating manager daemon smithi046.qrfdvr 2024-04-27T03:54:41.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:40 smithi046 ceph-mon[26899]: osdmap e2: 0 total, 0 up, 0 in 2024-04-27T03:54:41.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:40 smithi046 ceph-mon[26899]: mgrmap e6: smithi046.qrfdvr(active, starting, since 0.00324482s) 2024-04-27T03:54:41.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:40 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-04-27T03:54:41.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:40 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mgr metadata", "who": "smithi046.qrfdvr", "id": "smithi046.qrfdvr"}]: dispatch 2024-04-27T03:54:41.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:40 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T03:54:41.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:40 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T03:54:41.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:40 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T03:54:41.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:40 smithi046 ceph-mon[26899]: Manager daemon smithi046.qrfdvr is now available 2024-04-27T03:54:41.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:40 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:41.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:40 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:41.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:40 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:54:41.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:40 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:54:41.178 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:40 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.qrfdvr/mirror_snapshot_schedule"}]: dispatch 2024-04-27T03:54:41.178 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:40 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.qrfdvr/trash_purge_schedule"}]: dispatch 2024-04-27T03:54:41.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:41 smithi046 ceph-mon[26899]: Found migration_current of "None". Setting to last migration. 2024-04-27T03:54:41.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:41 smithi046 ceph-mon[26899]: mgrmap e7: smithi046.qrfdvr(active, since 1.00535s) 2024-04-27T03:54:42.231 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-04-27T03:54:42.231 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-27T03:54:42.231 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-27T03:54:42.232 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-04-27T03:54:42.232 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.802+0000 7fbab590b700 1 Processor -- start 2024-04-27T03:54:42.232 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.802+0000 7fbab590b700 1 -- start start 2024-04-27T03:54:42.232 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.802+0000 7fbab590b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0106a70 0x7fbab0106e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:42.232 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.802+0000 7fbab590b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab0107460 con 0x7fbab0106a70 2024-04-27T03:54:42.232 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.803+0000 7fbaaeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0106a70 0x7fbab0106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:42.232 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.803+0000 7fbaaeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0106a70 0x7fbab0106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58472/0 (socket says 172.21.15.46:58472) 2024-04-27T03:54:42.232 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.803+0000 7fbaaeffd700 1 -- 172.21.15.46:0/1464374380 learned_addr learned my addr 172.21.15.46:0/1464374380 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:42.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.803+0000 7fbaaeffd700 1 -- 172.21.15.46:0/1464374380 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab01056d0 con 0x7fbab0106a70 2024-04-27T03:54:42.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.803+0000 7fbaaeffd700 1 --2- 172.21.15.46:0/1464374380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0106a70 0x7fbab0106e90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbaa00098a0 tx=0x7fbaa0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb73efb56330ee9e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:42.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.803+0000 7fbaadffb700 1 -- 172.21.15.46:0/1464374380 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaa0004180 con 0x7fbab0106a70 2024-04-27T03:54:42.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.803+0000 7fbaadffb700 1 -- 172.21.15.46:0/1464374380 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fbaa00042e0 con 0x7fbab0106a70 2024-04-27T03:54:42.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.803+0000 7fbaadffb700 1 -- 172.21.15.46:0/1464374380 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaa0011a70 con 0x7fbab0106a70 2024-04-27T03:54:42.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.804+0000 7fbab590b700 1 -- 172.21.15.46:0/1464374380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0106a70 msgr2=0x7fbab0106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:42.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.804+0000 7fbab590b700 1 --2- 172.21.15.46:0/1464374380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0106a70 0x7fbab0106e90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbaa00098a0 tx=0x7fbaa0009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:42.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.804+0000 7fbab590b700 1 -- 172.21.15.46:0/1464374380 shutdown_connections 2024-04-27T03:54:42.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.804+0000 7fbab590b700 1 --2- 172.21.15.46:0/1464374380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0106a70 0x7fbab0106e90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:42.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.804+0000 7fbab590b700 1 -- 172.21.15.46:0/1464374380 >> 172.21.15.46:0/1464374380 conn(0x7fbab01001c0 msgr2=0x7fbab0102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:42.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.804+0000 7fbab590b700 1 -- 172.21.15.46:0/1464374380 shutdown_connections 2024-04-27T03:54:42.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.804+0000 7fbab590b700 1 -- 172.21.15.46:0/1464374380 wait complete. 2024-04-27T03:54:42.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.805+0000 7fbab590b700 1 Processor -- start 2024-04-27T03:54:42.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.805+0000 7fbab590b700 1 -- start start 2024-04-27T03:54:42.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.805+0000 7fbab590b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0106a70 0x7fbab019a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:42.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.805+0000 7fbab590b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab019a930 con 0x7fbab0106a70 2024-04-27T03:54:42.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.805+0000 7fbaaeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0106a70 0x7fbab019a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:42.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.805+0000 7fbaaeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0106a70 0x7fbab019a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58478/0 (socket says 172.21.15.46:58478) 2024-04-27T03:54:42.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.805+0000 7fbaaeffd700 1 -- 172.21.15.46:0/1937875240 learned_addr learned my addr 172.21.15.46:0/1937875240 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:42.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.805+0000 7fbaaeffd700 1 -- 172.21.15.46:0/1937875240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa0009550 con 0x7fbab0106a70 2024-04-27T03:54:42.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.806+0000 7fbaaeffd700 1 --2- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0106a70 0x7fbab019a3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0011a20 tx=0x7fbaa00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:42.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.806+0000 7fbab4909700 1 -- 172.21.15.46:0/1937875240 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaa0003b10 con 0x7fbab0106a70 2024-04-27T03:54:42.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.806+0000 7fbab590b700 1 -- 172.21.15.46:0/1937875240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab019ab30 con 0x7fbab0106a70 2024-04-27T03:54:42.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.806+0000 7fbab4909700 1 -- 172.21.15.46:0/1937875240 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fbaa001b8c0 con 0x7fbab0106a70 2024-04-27T03:54:42.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.806+0000 7fbab4909700 1 -- 172.21.15.46:0/1937875240 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaa00237b0 con 0x7fbab0106a70 2024-04-27T03:54:42.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.806+0000 7fbab590b700 1 -- 172.21.15.46:0/1937875240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab019afd0 con 0x7fbab0106a70 2024-04-27T03:54:42.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.807+0000 7fbab4909700 1 -- 172.21.15.46:0/1937875240 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fbaa0023a30 con 0x7fbab0106a70 2024-04-27T03:54:42.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.807+0000 7fbab4909700 1 --2- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fba9c0386e0 0x7fba9c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:42.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.807+0000 7fbab590b700 1 -- 172.21.15.46:0/1937875240 --> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbab0062510 con 0x7fba9c0386e0 2024-04-27T03:54:42.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.807+0000 7fbab4909700 1 -- 172.21.15.46:0/1937875240 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbaa004ca60 con 0x7fbab0106a70 2024-04-27T03:54:42.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.807+0000 7fbaae7fc700 1 -- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fba9c0386e0 msgr2=0x7fba9c03aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3824329935 2024-04-27T03:54:42.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:36.807+0000 7fbaae7fc700 1 --2- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fba9c0386e0 0x7fba9c03aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-27T03:54:42.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:37.008+0000 7fbaae7fc700 1 -- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fba9c0386e0 msgr2=0x7fba9c03aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3824329935 2024-04-27T03:54:42.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:37.008+0000 7fbaae7fc700 1 --2- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fba9c0386e0 0x7fba9c03aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-27T03:54:42.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:37.409+0000 7fbaae7fc700 1 -- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fba9c0386e0 msgr2=0x7fba9c03aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3824329935 2024-04-27T03:54:42.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:37.409+0000 7fbaae7fc700 1 --2- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fba9c0386e0 0x7fba9c03aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-27T03:54:42.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:38.210+0000 7fbaae7fc700 1 -- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fba9c0386e0 msgr2=0x7fba9c03aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3824329935 2024-04-27T03:54:42.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:38.210+0000 7fbaae7fc700 1 --2- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fba9c0386e0 0x7fba9c03aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-27T03:54:42.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:39.811+0000 7fbaae7fc700 1 -- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fba9c0386e0 msgr2=0x7fba9c03aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3824329935 2024-04-27T03:54:42.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:39.811+0000 7fbaae7fc700 1 --2- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fba9c0386e0 0x7fba9c03aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-27T03:54:42.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:40.660+0000 7fbab4909700 1 -- 172.21.15.46:0/1937875240 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mgrmap(e 6) v1 ==== 45323+0+0 (secure 0 0 0) 0x7fbaa002cc90 con 0x7fbab0106a70 2024-04-27T03:54:42.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:40.661+0000 7fbab4909700 1 -- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fba9c0386e0 msgr2=0x7fba9c03aba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T03:54:42.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:40.661+0000 7fbab4909700 1 --2- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fba9c0386e0 0x7fba9c03aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:42.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.662+0000 7fbab4909700 1 -- 172.21.15.46:0/1937875240 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 7) v1 ==== 45450+0+0 (secure 0 0 0) 0x7fbaa004e0c0 con 0x7fbab0106a70 2024-04-27T03:54:42.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.662+0000 7fbab4909700 1 --2- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fba9c03beb0 0x7fba9c03e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:42.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.662+0000 7fbab4909700 1 -- 172.21.15.46:0/1937875240 --> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbab0062510 con 0x7fba9c03beb0 2024-04-27T03:54:42.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.663+0000 7fbaae7fc700 1 --2- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fba9c03beb0 0x7fba9c03e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:42.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.663+0000 7fbaae7fc700 1 --2- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fba9c03beb0 0x7fba9c03e2a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fba98003a10 tx=0x7fba98009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:42.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.664+0000 7fbab4909700 1 -- 172.21.15.46:0/1937875240 <== mgr.14118 v2:172.21.15.46:6800/333567692 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fbab0062510 con 0x7fba9c03beb0 2024-04-27T03:54:42.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.670+0000 7fbab590b700 1 -- 172.21.15.46:0/1937875240 --> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fbab0109910 con 0x7fba9c03beb0 2024-04-27T03:54:42.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.670+0000 7fbab4909700 1 -- 172.21.15.46:0/1937875240 <== mgr.14118 v2:172.21.15.46:6800/333567692 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fbab0109910 con 0x7fba9c03beb0 2024-04-27T03:54:42.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.671+0000 7fbab590b700 1 -- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fba9c03beb0 msgr2=0x7fba9c03e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:42.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.671+0000 7fbab590b700 1 --2- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fba9c03beb0 0x7fba9c03e2a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fba98003a10 tx=0x7fba98009290 comp rx=0 tx=0).stop 2024-04-27T03:54:42.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.671+0000 7fbab590b700 1 -- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0106a70 msgr2=0x7fbab019a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:42.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.671+0000 7fbab590b700 1 --2- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0106a70 0x7fbab019a3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0011a20 tx=0x7fbaa00047c0 comp rx=0 tx=0).stop 2024-04-27T03:54:42.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.671+0000 7fbab590b700 1 -- 172.21.15.46:0/1937875240 shutdown_connections 2024-04-27T03:54:42.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.671+0000 7fbab590b700 1 --2- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fba9c03beb0 0x7fba9c03e2a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:42.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.671+0000 7fbab590b700 1 --2- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0106a70 0x7fbab019a3f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:42.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.671+0000 7fbab590b700 1 --2- 172.21.15.46:0/1937875240 >> [v2:172.21.15.46:6800/3824329935,v1:172.21.15.46:6801/3824329935] conn(0x7fba9c0386e0 0x7fba9c03aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:42.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.671+0000 7fbab590b700 1 -- 172.21.15.46:0/1937875240 >> 172.21.15.46:0/1937875240 conn(0x7fbab01001c0 msgr2=0x7fbab0101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:42.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.671+0000 7fbab590b700 1 -- 172.21.15.46:0/1937875240 shutdown_connections 2024-04-27T03:54:42.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:41.671+0000 7fbab590b700 1 -- 172.21.15.46:0/1937875240 wait complete. 2024-04-27T03:54:42.244 INFO:teuthology.orchestra.run.smithi046.stdout:mgr epoch 5 is available 2024-04-27T03:54:42.244 INFO:teuthology.orchestra.run.smithi046.stdout:Setting orchestrator backend to cephadm... 2024-04-27T03:54:43.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:42 smithi046 ceph-mon[26899]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-27T03:54:43.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:42 smithi046 ceph-mon[26899]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-27T03:54:43.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:42 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:43.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:42 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:43.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:42 smithi046 ceph-mon[26899]: [27/Apr/2024:03:54:42] ENGINE Bus STARTING 2024-04-27T03:54:43.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:42 smithi046 ceph-mon[26899]: [27/Apr/2024:03:54:42] ENGINE Serving on http://172.21.15.46:8765 2024-04-27T03:54:43.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:42 smithi046 ceph-mon[26899]: [27/Apr/2024:03:54:42] ENGINE Serving on https://172.21.15.46:7150 2024-04-27T03:54:43.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:42 smithi046 ceph-mon[26899]: [27/Apr/2024:03:54:42] ENGINE Bus STARTED 2024-04-27T03:54:43.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:42 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:54:43.499 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.692+0000 7f30b9d91700 1 Processor -- start 2024-04-27T03:54:43.500 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.692+0000 7f30b9d91700 1 -- start start 2024-04-27T03:54:43.500 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.692+0000 7f30b9d91700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30b4107030 0x7f30b4109420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:43.500 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.692+0000 7f30b9d91700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b4079ed0 con 0x7f30b4107030 2024-04-27T03:54:43.500 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.692+0000 7f30b37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30b4107030 0x7f30b4109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:43.500 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.692+0000 7f30b37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30b4107030 0x7f30b4109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34580/0 (socket says 172.21.15.46:34580) 2024-04-27T03:54:43.501 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.692+0000 7f30b37fe700 1 -- 172.21.15.46:0/1472476928 learned_addr learned my addr 172.21.15.46:0/1472476928 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:43.501 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.693+0000 7f30b37fe700 1 -- 172.21.15.46:0/1472476928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b4109960 con 0x7f30b4107030 2024-04-27T03:54:43.501 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.693+0000 7f30b37fe700 1 --2- 172.21.15.46:0/1472476928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30b4107030 0x7f30b4109420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0098a0 tx=0x7f30ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e800051512a2cff server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:43.501 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.693+0000 7f30b27fc700 1 -- 172.21.15.46:0/1472476928 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30ac004180 con 0x7f30b4107030 2024-04-27T03:54:43.501 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.693+0000 7f30b27fc700 1 -- 172.21.15.46:0/1472476928 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f30ac0042e0 con 0x7f30b4107030 2024-04-27T03:54:43.501 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.693+0000 7f30b27fc700 1 -- 172.21.15.46:0/1472476928 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30ac011a70 con 0x7f30b4107030 2024-04-27T03:54:43.501 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.694+0000 7f30b9d91700 1 -- 172.21.15.46:0/1472476928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30b4107030 msgr2=0x7f30b4109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:43.501 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.694+0000 7f30b9d91700 1 --2- 172.21.15.46:0/1472476928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30b4107030 0x7f30b4109420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0098a0 tx=0x7f30ac009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:43.501 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.694+0000 7f30b9d91700 1 -- 172.21.15.46:0/1472476928 shutdown_connections 2024-04-27T03:54:43.502 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.694+0000 7f30b9d91700 1 --2- 172.21.15.46:0/1472476928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30b4107030 0x7f30b4109420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:43.502 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.694+0000 7f30b9d91700 1 -- 172.21.15.46:0/1472476928 >> 172.21.15.46:0/1472476928 conn(0x7f30b4100ae0 msgr2=0x7f30b4102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:43.502 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.694+0000 7f30b9d91700 1 -- 172.21.15.46:0/1472476928 shutdown_connections 2024-04-27T03:54:43.502 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.694+0000 7f30b9d91700 1 -- 172.21.15.46:0/1472476928 wait complete. 2024-04-27T03:54:43.502 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.694+0000 7f30b9d91700 1 Processor -- start 2024-04-27T03:54:43.502 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.695+0000 7f30b9d91700 1 -- start start 2024-04-27T03:54:43.502 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.695+0000 7f30b9d91700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30b4107030 0x7f30b4106860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:43.502 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.695+0000 7f30b9d91700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b4106da0 con 0x7f30b4107030 2024-04-27T03:54:43.502 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.695+0000 7f30b37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30b4107030 0x7f30b4106860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:43.503 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.695+0000 7f30b37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30b4107030 0x7f30b4106860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34588/0 (socket says 172.21.15.46:34588) 2024-04-27T03:54:43.503 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.695+0000 7f30b37fe700 1 -- 172.21.15.46:0/2351456166 learned_addr learned my addr 172.21.15.46:0/2351456166 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:43.503 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.695+0000 7f30b37fe700 1 -- 172.21.15.46:0/2351456166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac009550 con 0x7f30b4107030 2024-04-27T03:54:43.503 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.695+0000 7f30b37fe700 1 --2- 172.21.15.46:0/2351456166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30b4107030 0x7f30b4106860 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f30ac011a20 tx=0x7f30ac0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:43.503 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.695+0000 7f30b0ff9700 1 -- 172.21.15.46:0/2351456166 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30ac003b10 con 0x7f30b4107030 2024-04-27T03:54:43.503 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.695+0000 7f30b0ff9700 1 -- 172.21.15.46:0/2351456166 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f30ac01b8c0 con 0x7f30b4107030 2024-04-27T03:54:43.503 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.695+0000 7f30b9d91700 1 -- 172.21.15.46:0/2351456166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b4104fe0 con 0x7f30b4107030 2024-04-27T03:54:43.503 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.695+0000 7f30b0ff9700 1 -- 172.21.15.46:0/2351456166 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30ac0237b0 con 0x7f30b4107030 2024-04-27T03:54:43.503 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.695+0000 7f30b9d91700 1 -- 172.21.15.46:0/2351456166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30b4105480 con 0x7f30b4107030 2024-04-27T03:54:43.503 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.696+0000 7f30b0ff9700 1 -- 172.21.15.46:0/2351456166 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 7) v1 ==== 45450+0+0 (secure 0 0 0) 0x7f30ac01b440 con 0x7f30b4107030 2024-04-27T03:54:43.504 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.696+0000 7f30b0ff9700 1 --2- 172.21.15.46:0/2351456166 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f30a0038610 0x7f30a003aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:43.504 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.696+0000 7f30b9d91700 1 -- 172.21.15.46:0/2351456166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30b410a6e0 con 0x7f30b4107030 2024-04-27T03:54:43.504 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.697+0000 7f30b0ff9700 1 -- 172.21.15.46:0/2351456166 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f30ac04c870 con 0x7f30b4107030 2024-04-27T03:54:43.504 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.697+0000 7f30b2ffd700 1 --2- 172.21.15.46:0/2351456166 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f30a0038610 0x7f30a003aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:43.504 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.697+0000 7f30b2ffd700 1 --2- 172.21.15.46:0/2351456166 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f30a0038610 0x7f30a003aad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f309c006fd0 tx=0x7f309c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:43.504 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.702+0000 7f30b0ff9700 1 -- 172.21.15.46:0/2351456166 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f30ac02c3f0 con 0x7f30b4107030 2024-04-27T03:54:43.504 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.864+0000 7f30b0ff9700 1 -- 172.21.15.46:0/2351456166 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f30ac02c770 con 0x7f30b4107030 2024-04-27T03:54:43.504 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.924+0000 7f30b9d91700 1 -- 172.21.15.46:0/2351456166 --> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f30b4105c30 con 0x7f30a0038610 2024-04-27T03:54:43.504 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.932+0000 7f30b0ff9700 1 -- 172.21.15.46:0/2351456166 <== mgr.14118 v2:172.21.15.46:6800/333567692 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f30b4105c30 con 0x7f30a0038610 2024-04-27T03:54:43.504 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.934+0000 7f30b9d91700 1 -- 172.21.15.46:0/2351456166 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f30a0038610 msgr2=0x7f30a003aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:43.504 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.934+0000 7f30b9d91700 1 --2- 172.21.15.46:0/2351456166 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f30a0038610 0x7f30a003aad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f309c006fd0 tx=0x7f309c006e40 comp rx=0 tx=0).stop 2024-04-27T03:54:43.504 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.934+0000 7f30b9d91700 1 -- 172.21.15.46:0/2351456166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30b4107030 msgr2=0x7f30b4106860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:43.504 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.934+0000 7f30b9d91700 1 --2- 172.21.15.46:0/2351456166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30b4107030 0x7f30b4106860 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f30ac011a20 tx=0x7f30ac0047c0 comp rx=0 tx=0).stop 2024-04-27T03:54:43.505 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.934+0000 7f30b9d91700 1 -- 172.21.15.46:0/2351456166 shutdown_connections 2024-04-27T03:54:43.505 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.934+0000 7f30b9d91700 1 --2- 172.21.15.46:0/2351456166 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f30a0038610 0x7f30a003aad0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:43.505 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.934+0000 7f30b9d91700 1 --2- 172.21.15.46:0/2351456166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30b4107030 0x7f30b4106860 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:43.505 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.934+0000 7f30b9d91700 1 -- 172.21.15.46:0/2351456166 >> 172.21.15.46:0/2351456166 conn(0x7f30b4100ae0 msgr2=0x7f30b41015b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:43.505 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.934+0000 7f30b9d91700 1 -- 172.21.15.46:0/2351456166 shutdown_connections 2024-04-27T03:54:43.505 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:42.934+0000 7f30b9d91700 1 -- 172.21.15.46:0/2351456166 wait complete. 2024-04-27T03:54:43.899 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:43 smithi046 ceph-mon[26899]: mgrmap e8: smithi046.qrfdvr(active, since 2s) 2024-04-27T03:54:43.899 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:43 smithi046 ceph-mon[26899]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:54:43.899 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:43 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:43.899 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:43 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:54:44.644 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-27T03:54:44.645 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.896+0000 7fadbe978700 1 Processor -- start 2024-04-27T03:54:44.645 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.896+0000 7fadbe978700 1 -- start start 2024-04-27T03:54:44.645 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.896+0000 7fadbe978700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb81065c0 0x7fadb81069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:44.645 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.896+0000 7fadbe978700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb8106fb0 con 0x7fadb81065c0 2024-04-27T03:54:44.645 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.896+0000 7fadb7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb81065c0 0x7fadb81069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:44.645 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.896+0000 7fadb7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb81065c0 0x7fadb81069e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34600/0 (socket says 172.21.15.46:34600) 2024-04-27T03:54:44.645 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.896+0000 7fadb7fff700 1 -- 172.21.15.46:0/2966260839 learned_addr learned my addr 172.21.15.46:0/2966260839 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:44.645 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.896+0000 7fadb7fff700 1 -- 172.21.15.46:0/2966260839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb81077c0 con 0x7fadb81065c0 2024-04-27T03:54:44.645 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.897+0000 7fadb7fff700 1 --2- 172.21.15.46:0/2966260839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb81065c0 0x7fadb81069e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fada80098a0 tx=0x7fada8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=700cae30e0cf2858 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:44.645 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.897+0000 7fadb6ffd700 1 -- 172.21.15.46:0/2966260839 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fada8004180 con 0x7fadb81065c0 2024-04-27T03:54:44.645 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.897+0000 7fadb6ffd700 1 -- 172.21.15.46:0/2966260839 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fada80042e0 con 0x7fadb81065c0 2024-04-27T03:54:44.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.897+0000 7fadb6ffd700 1 -- 172.21.15.46:0/2966260839 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fada8011a70 con 0x7fadb81065c0 2024-04-27T03:54:44.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.897+0000 7fadbe978700 1 -- 172.21.15.46:0/2966260839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb81065c0 msgr2=0x7fadb81069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:44.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.897+0000 7fadbe978700 1 --2- 172.21.15.46:0/2966260839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb81065c0 0x7fadb81069e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fada80098a0 tx=0x7fada8009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:44.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.897+0000 7fadbe978700 1 -- 172.21.15.46:0/2966260839 shutdown_connections 2024-04-27T03:54:44.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.897+0000 7fadbe978700 1 --2- 172.21.15.46:0/2966260839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb81065c0 0x7fadb81069e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:44.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.897+0000 7fadbe978700 1 -- 172.21.15.46:0/2966260839 >> 172.21.15.46:0/2966260839 conn(0x7fadb8101be0 msgr2=0x7fadb8104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:44.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.898+0000 7fadbe978700 1 -- 172.21.15.46:0/2966260839 shutdown_connections 2024-04-27T03:54:44.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.898+0000 7fadbe978700 1 -- 172.21.15.46:0/2966260839 wait complete. 2024-04-27T03:54:44.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.898+0000 7fadbe978700 1 Processor -- start 2024-04-27T03:54:44.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.898+0000 7fadbe978700 1 -- start start 2024-04-27T03:54:44.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.899+0000 7fadbe978700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb81065c0 0x7fadb819a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:44.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.899+0000 7fadbe978700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb819a830 con 0x7fadb81065c0 2024-04-27T03:54:44.646 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.899+0000 7fadb7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb81065c0 0x7fadb819a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:44.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.899+0000 7fadb7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb81065c0 0x7fadb819a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34604/0 (socket says 172.21.15.46:34604) 2024-04-27T03:54:44.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.899+0000 7fadb7fff700 1 -- 172.21.15.46:0/1359410299 learned_addr learned my addr 172.21.15.46:0/1359410299 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:44.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.899+0000 7fadb7fff700 1 -- 172.21.15.46:0/1359410299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada8009550 con 0x7fadb81065c0 2024-04-27T03:54:44.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.899+0000 7fadb7fff700 1 --2- 172.21.15.46:0/1359410299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb81065c0 0x7fadb819a2f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fada8011710 tx=0x7fada80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:44.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.899+0000 7fadb57fa700 1 -- 172.21.15.46:0/1359410299 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fada8003b10 con 0x7fadb81065c0 2024-04-27T03:54:44.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.899+0000 7fadb57fa700 1 -- 172.21.15.46:0/1359410299 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fada801b8c0 con 0x7fadb81065c0 2024-04-27T03:54:44.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.899+0000 7fadbe978700 1 -- 172.21.15.46:0/1359410299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb819aa30 con 0x7fadb81065c0 2024-04-27T03:54:44.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.899+0000 7fadb57fa700 1 -- 172.21.15.46:0/1359410299 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fada8019770 con 0x7fadb81065c0 2024-04-27T03:54:44.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.900+0000 7fadbe978700 1 -- 172.21.15.46:0/1359410299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb819aed0 con 0x7fadb81065c0 2024-04-27T03:54:44.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.900+0000 7fadb57fa700 1 -- 172.21.15.46:0/1359410299 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fada80199f0 con 0x7fadb81065c0 2024-04-27T03:54:44.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.901+0000 7fadbe978700 1 -- 172.21.15.46:0/1359410299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb810a9f0 con 0x7fadb81065c0 2024-04-27T03:54:44.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.901+0000 7fadb57fa700 1 --2- 172.21.15.46:0/1359410299 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fada0038720 0x7fada003abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:44.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.901+0000 7fadb57fa700 1 -- 172.21.15.46:0/1359410299 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fada804ca80 con 0x7fadb81065c0 2024-04-27T03:54:44.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.901+0000 7fadb77fe700 1 --2- 172.21.15.46:0/1359410299 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fada0038720 0x7fada003abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:44.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.901+0000 7fadb77fe700 1 --2- 172.21.15.46:0/1359410299 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fada0038720 0x7fada003abe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fadac006fd0 tx=0x7fadac006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:44.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:43.906+0000 7fadb57fa700 1 -- 172.21.15.46:0/1359410299 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fada8018530 con 0x7fadb81065c0 2024-04-27T03:54:44.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:44.108+0000 7fadbe978700 1 -- 172.21.15.46:0/1359410299 --> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fadb8102e50 con 0x7fada0038720 2024-04-27T03:54:44.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:44.109+0000 7fadb57fa700 1 -- 172.21.15.46:0/1359410299 <== mgr.14118 v2:172.21.15.46:6800/333567692 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fadb8102e50 con 0x7fada0038720 2024-04-27T03:54:44.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:44.111+0000 7fadbe978700 1 -- 172.21.15.46:0/1359410299 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fada0038720 msgr2=0x7fada003abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:44.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:44.111+0000 7fadbe978700 1 --2- 172.21.15.46:0/1359410299 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fada0038720 0x7fada003abe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fadac006fd0 tx=0x7fadac006e40 comp rx=0 tx=0).stop 2024-04-27T03:54:44.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:44.111+0000 7fadbe978700 1 -- 172.21.15.46:0/1359410299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb81065c0 msgr2=0x7fadb819a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:44.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:44.111+0000 7fadbe978700 1 --2- 172.21.15.46:0/1359410299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb81065c0 0x7fadb819a2f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fada8011710 tx=0x7fada80047c0 comp rx=0 tx=0).stop 2024-04-27T03:54:44.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:44.111+0000 7fadbe978700 1 -- 172.21.15.46:0/1359410299 shutdown_connections 2024-04-27T03:54:44.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:44.111+0000 7fadbe978700 1 --2- 172.21.15.46:0/1359410299 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fada0038720 0x7fada003abe0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:44.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:44.111+0000 7fadbe978700 1 --2- 172.21.15.46:0/1359410299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb81065c0 0x7fadb819a2f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:44.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:44.111+0000 7fadbe978700 1 -- 172.21.15.46:0/1359410299 >> 172.21.15.46:0/1359410299 conn(0x7fadb8101be0 msgr2=0x7fadb8102740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:44.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:44.111+0000 7fadbe978700 1 -- 172.21.15.46:0/1359410299 shutdown_connections 2024-04-27T03:54:44.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:44.111+0000 7fadbe978700 1 -- 172.21.15.46:0/1359410299 wait complete. 2024-04-27T03:54:44.650 INFO:teuthology.orchestra.run.smithi046.stdout:Generating ssh key... 2024-04-27T03:54:45.080 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:44 smithi046 ceph-mon[26899]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:54:46.125 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.113+0000 7fa219b71700 1 Processor -- start 2024-04-27T03:54:46.126 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.113+0000 7fa219b71700 1 -- start start 2024-04-27T03:54:46.126 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.113+0000 7fa219b71700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa214108800 0x7fa214108c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:46.126 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.113+0000 7fa219b71700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2141091f0 con 0x7fa214108800 2024-04-27T03:54:46.126 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.113+0000 7fa2137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa214108800 0x7fa214108c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:46.126 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.113+0000 7fa2137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa214108800 0x7fa214108c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34606/0 (socket says 172.21.15.46:34606) 2024-04-27T03:54:46.126 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.113+0000 7fa2137fe700 1 -- 172.21.15.46:0/3629799762 learned_addr learned my addr 172.21.15.46:0/3629799762 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:46.126 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.114+0000 7fa2137fe700 1 -- 172.21.15.46:0/3629799762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2141099b0 con 0x7fa214108800 2024-04-27T03:54:46.126 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.114+0000 7fa2137fe700 1 --2- 172.21.15.46:0/3629799762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa214108800 0x7fa214108c20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa204009e80 tx=0x7fa20400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d173413205b0d5c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:46.126 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.114+0000 7fa2127fc700 1 -- 172.21.15.46:0/3629799762 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa20400e070 con 0x7fa214108800 2024-04-27T03:54:46.126 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.114+0000 7fa2127fc700 1 -- 172.21.15.46:0/3629799762 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa20400b8c0 con 0x7fa214108800 2024-04-27T03:54:46.127 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.114+0000 7fa2127fc700 1 -- 172.21.15.46:0/3629799762 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa204004440 con 0x7fa214108800 2024-04-27T03:54:46.127 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.115+0000 7fa219b71700 1 -- 172.21.15.46:0/3629799762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa214108800 msgr2=0x7fa214108c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:46.127 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.115+0000 7fa219b71700 1 --2- 172.21.15.46:0/3629799762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa214108800 0x7fa214108c20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa204009e80 tx=0x7fa20400b330 comp rx=0 tx=0).stop 2024-04-27T03:54:46.127 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.115+0000 7fa219b71700 1 -- 172.21.15.46:0/3629799762 shutdown_connections 2024-04-27T03:54:46.127 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.115+0000 7fa219b71700 1 --2- 172.21.15.46:0/3629799762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa214108800 0x7fa214108c20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:46.127 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.115+0000 7fa219b71700 1 -- 172.21.15.46:0/3629799762 >> 172.21.15.46:0/3629799762 conn(0x7fa21407bd90 msgr2=0x7fa2141064f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:46.127 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.115+0000 7fa219b71700 1 -- 172.21.15.46:0/3629799762 shutdown_connections 2024-04-27T03:54:46.127 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.115+0000 7fa219b71700 1 -- 172.21.15.46:0/3629799762 wait complete. 2024-04-27T03:54:46.127 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.115+0000 7fa219b71700 1 Processor -- start 2024-04-27T03:54:46.127 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.116+0000 7fa219b71700 1 -- start start 2024-04-27T03:54:46.127 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.116+0000 7fa219b71700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa214108800 0x7fa21419c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:46.127 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.116+0000 7fa219b71700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21419c9c0 con 0x7fa214108800 2024-04-27T03:54:46.127 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.116+0000 7fa2137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa214108800 0x7fa21419c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:46.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.116+0000 7fa2137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa214108800 0x7fa21419c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34620/0 (socket says 172.21.15.46:34620) 2024-04-27T03:54:46.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.116+0000 7fa2137fe700 1 -- 172.21.15.46:0/1341366279 learned_addr learned my addr 172.21.15.46:0/1341366279 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:46.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.116+0000 7fa2137fe700 1 -- 172.21.15.46:0/1341366279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa204009b30 con 0x7fa214108800 2024-04-27T03:54:46.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.116+0000 7fa2137fe700 1 --2- 172.21.15.46:0/1341366279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa214108800 0x7fa21419c480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa204003c30 tx=0x7fa204003d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:46.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.116+0000 7fa210ff9700 1 -- 172.21.15.46:0/1341366279 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa20400e040 con 0x7fa214108800 2024-04-27T03:54:46.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.116+0000 7fa210ff9700 1 -- 172.21.15.46:0/1341366279 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa20400bc40 con 0x7fa214108800 2024-04-27T03:54:46.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.116+0000 7fa219b71700 1 -- 172.21.15.46:0/1341366279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa21419cbc0 con 0x7fa214108800 2024-04-27T03:54:46.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.116+0000 7fa210ff9700 1 -- 172.21.15.46:0/1341366279 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa204012750 con 0x7fa214108800 2024-04-27T03:54:46.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.117+0000 7fa219b71700 1 -- 172.21.15.46:0/1341366279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa21419d0c0 con 0x7fa214108800 2024-04-27T03:54:46.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.117+0000 7fa210ff9700 1 -- 172.21.15.46:0/1341366279 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fa204019040 con 0x7fa214108800 2024-04-27T03:54:46.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.118+0000 7fa219b71700 1 -- 172.21.15.46:0/1341366279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa21410cbe0 con 0x7fa214108800 2024-04-27T03:54:46.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.118+0000 7fa210ff9700 1 --2- 172.21.15.46:0/1341366279 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa2000386e0 0x7fa20003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:46.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.118+0000 7fa210ff9700 1 -- 172.21.15.46:0/1341366279 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa20404c1d0 con 0x7fa214108800 2024-04-27T03:54:46.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.118+0000 7fa212ffd700 1 --2- 172.21.15.46:0/1341366279 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa2000386e0 0x7fa20003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:46.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.118+0000 7fa212ffd700 1 --2- 172.21.15.46:0/1341366279 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa2000386e0 0x7fa20003aba0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc006fd0 tx=0x7fa1fc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:46.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.123+0000 7fa210ff9700 1 -- 172.21.15.46:0/1341366279 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fa204012a10 con 0x7fa214108800 2024-04-27T03:54:46.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.328+0000 7fa219b71700 1 -- 172.21.15.46:0/1341366279 --> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2141062f0 con 0x7fa2000386e0 2024-04-27T03:54:46.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.576+0000 7fa210ff9700 1 -- 172.21.15.46:0/1341366279 <== mgr.14118 v2:172.21.15.46:6800/333567692 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa2141062f0 con 0x7fa2000386e0 2024-04-27T03:54:46.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.578+0000 7fa219b71700 1 -- 172.21.15.46:0/1341366279 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa2000386e0 msgr2=0x7fa20003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:46.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.578+0000 7fa219b71700 1 --2- 172.21.15.46:0/1341366279 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa2000386e0 0x7fa20003aba0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc006fd0 tx=0x7fa1fc006e40 comp rx=0 tx=0).stop 2024-04-27T03:54:46.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.578+0000 7fa219b71700 1 -- 172.21.15.46:0/1341366279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa214108800 msgr2=0x7fa21419c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:46.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.578+0000 7fa219b71700 1 --2- 172.21.15.46:0/1341366279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa214108800 0x7fa21419c480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa204003c30 tx=0x7fa204003d10 comp rx=0 tx=0).stop 2024-04-27T03:54:46.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.578+0000 7fa219b71700 1 -- 172.21.15.46:0/1341366279 shutdown_connections 2024-04-27T03:54:46.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.578+0000 7fa219b71700 1 --2- 172.21.15.46:0/1341366279 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa2000386e0 0x7fa20003aba0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:46.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.578+0000 7fa219b71700 1 --2- 172.21.15.46:0/1341366279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa214108800 0x7fa21419c480 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:46.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.578+0000 7fa219b71700 1 -- 172.21.15.46:0/1341366279 >> 172.21.15.46:0/1341366279 conn(0x7fa21407bd90 msgr2=0x7fa214105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:46.130 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.578+0000 7fa219b71700 1 -- 172.21.15.46:0/1341366279 shutdown_connections 2024-04-27T03:54:46.130 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:45.578+0000 7fa219b71700 1 -- 172.21.15.46:0/1341366279 wait complete. 2024-04-27T03:54:46.758 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:46 smithi046 ceph-mon[26899]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:54:46.759 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:46 smithi046 ceph-mon[26899]: Generating ssh key... 2024-04-27T03:54:46.759 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:46 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:46.759 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:46 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:47.275 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCweCM9Acgf13VvpFVvlKocoNJjU3vbp+GUML+rGDel6vYoVSAquz08wg/kdr7Mqr+5MQ2k7tdZ3515qvjm+JXC6DAq8gj0TV9igHR+sNSCCGqNSr14AOqiEbmCkGIeP0bOUnoUqipcvqXXozZ1Ed7hFxVxcwODpSX9qaqhuyV+ZsQhGT8mboHSqpcgnS2z/HBIh7Srd3UaY0d1p/bAETk8K+aymS/zS6wtbNA5YM9jfBIaQ1tnYcPii30jC7BFg6UxRtg5j+sfTAf3b67jBm7n51YRLA1ZL/y4cu2qpzYRoN6u0dCSmOzdbBiFylTKSj0IOvrhmGSqNZ30hfGMuHEAGQuQ13S79eUaP9n0ssqFDcUXp3z3rqvFXNzymxP41i94Wc61bc0sYvqC863l6ixgBXTR3cDXD6vMz0y/dlwhHvo5z9yqEerxwRdt0+fmgkpZITSAVU3cEBYIOd3JJOBuzEU/MKKsa/n1g7ZLyYKsWDqhRUCUHp8WAZ7PLiO6Dz8= ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968 2024-04-27T03:54:47.275 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.528+0000 7f3679042700 1 Processor -- start 2024-04-27T03:54:47.275 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.528+0000 7f3679042700 1 -- start start 2024-04-27T03:54:47.276 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.528+0000 7f3679042700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36741065c0 0x7f36741069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:47.276 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.528+0000 7f3679042700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3674106fb0 con 0x7f36741065c0 2024-04-27T03:54:47.276 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.528+0000 7f3672d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36741065c0 0x7f36741069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:47.276 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.528+0000 7f3672d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36741065c0 0x7f36741069e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34622/0 (socket says 172.21.15.46:34622) 2024-04-27T03:54:47.276 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.528+0000 7f3672d9d700 1 -- 172.21.15.46:0/3345760651 learned_addr learned my addr 172.21.15.46:0/3345760651 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:47.276 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.528+0000 7f3672d9d700 1 -- 172.21.15.46:0/3345760651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36741077c0 con 0x7f36741065c0 2024-04-27T03:54:47.276 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.529+0000 7f3672d9d700 1 --2- 172.21.15.46:0/3345760651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36741065c0 0x7f36741069e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f36640098a0 tx=0x7f3664009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aecc4f7113c03ba server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:47.276 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.529+0000 7f3671d9b700 1 -- 172.21.15.46:0/3345760651 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3664004180 con 0x7f36741065c0 2024-04-27T03:54:47.276 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.529+0000 7f3671d9b700 1 -- 172.21.15.46:0/3345760651 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f36640042e0 con 0x7f36741065c0 2024-04-27T03:54:47.276 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.529+0000 7f3671d9b700 1 -- 172.21.15.46:0/3345760651 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3664011a70 con 0x7f36741065c0 2024-04-27T03:54:47.276 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.529+0000 7f3679042700 1 -- 172.21.15.46:0/3345760651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36741065c0 msgr2=0x7f36741069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:47.277 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.529+0000 7f3679042700 1 --2- 172.21.15.46:0/3345760651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36741065c0 0x7f36741069e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f36640098a0 tx=0x7f3664009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:47.277 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.529+0000 7f3679042700 1 -- 172.21.15.46:0/3345760651 shutdown_connections 2024-04-27T03:54:47.277 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.529+0000 7f3679042700 1 --2- 172.21.15.46:0/3345760651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36741065c0 0x7f36741069e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:47.277 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.529+0000 7f3679042700 1 -- 172.21.15.46:0/3345760651 >> 172.21.15.46:0/3345760651 conn(0x7f3674101be0 msgr2=0x7f3674104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:47.277 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.529+0000 7f3679042700 1 -- 172.21.15.46:0/3345760651 shutdown_connections 2024-04-27T03:54:47.277 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.529+0000 7f3679042700 1 -- 172.21.15.46:0/3345760651 wait complete. 2024-04-27T03:54:47.277 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.530+0000 7f3679042700 1 Processor -- start 2024-04-27T03:54:47.277 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.530+0000 7f3679042700 1 -- start start 2024-04-27T03:54:47.277 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.530+0000 7f3679042700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36741065c0 0x7f367419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:47.277 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.530+0000 7f3679042700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367419a830 con 0x7f36741065c0 2024-04-27T03:54:47.277 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.531+0000 7f3672d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36741065c0 0x7f367419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:47.277 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.531+0000 7f3672d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36741065c0 0x7f367419a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34638/0 (socket says 172.21.15.46:34638) 2024-04-27T03:54:47.278 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.531+0000 7f3672d9d700 1 -- 172.21.15.46:0/462076723 learned_addr learned my addr 172.21.15.46:0/462076723 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:47.278 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.531+0000 7f3672d9d700 1 -- 172.21.15.46:0/462076723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3664009550 con 0x7f36741065c0 2024-04-27T03:54:47.278 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.531+0000 7f3672d9d700 1 --2- 172.21.15.46:0/462076723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36741065c0 0x7f367419a2f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3664011710 tx=0x7f36640047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:47.278 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.531+0000 7f365bfff700 1 -- 172.21.15.46:0/462076723 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3664003b10 con 0x7f36741065c0 2024-04-27T03:54:47.278 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.531+0000 7f3679042700 1 -- 172.21.15.46:0/462076723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f367419aa30 con 0x7f36741065c0 2024-04-27T03:54:47.278 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.531+0000 7f365bfff700 1 -- 172.21.15.46:0/462076723 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f366401b8c0 con 0x7f36741065c0 2024-04-27T03:54:47.278 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.531+0000 7f365bfff700 1 -- 172.21.15.46:0/462076723 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3664019770 con 0x7f36741065c0 2024-04-27T03:54:47.278 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.531+0000 7f3679042700 1 -- 172.21.15.46:0/462076723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f367419aed0 con 0x7f36741065c0 2024-04-27T03:54:47.278 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.532+0000 7f365bfff700 1 -- 172.21.15.46:0/462076723 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f36640199f0 con 0x7f36741065c0 2024-04-27T03:54:47.278 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.532+0000 7f3679042700 1 -- 172.21.15.46:0/462076723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367410a9f0 con 0x7f36741065c0 2024-04-27T03:54:47.279 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.532+0000 7f365bfff700 1 --2- 172.21.15.46:0/462076723 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f365c038720 0x7f365c03abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:47.279 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.532+0000 7f365bfff700 1 -- 172.21.15.46:0/462076723 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f366404c9e0 con 0x7f36741065c0 2024-04-27T03:54:47.279 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.532+0000 7f367259c700 1 --2- 172.21.15.46:0/462076723 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f365c038720 0x7f365c03abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:47.279 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.533+0000 7f367259c700 1 --2- 172.21.15.46:0/462076723 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f365c038720 0x7f365c03abe0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3668006fd0 tx=0x7f3668006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:47.279 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.537+0000 7f365bfff700 1 -- 172.21.15.46:0/462076723 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f36640189a0 con 0x7f36741065c0 2024-04-27T03:54:47.279 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.742+0000 7f3679042700 1 -- 172.21.15.46:0/462076723 --> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f367419d7c0 con 0x7f365c038720 2024-04-27T03:54:47.279 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.742+0000 7f365bfff700 1 -- 172.21.15.46:0/462076723 <== mgr.14118 v2:172.21.15.46:6800/333567692 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f367419d7c0 con 0x7f365c038720 2024-04-27T03:54:47.279 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.744+0000 7f3679042700 1 -- 172.21.15.46:0/462076723 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f365c038720 msgr2=0x7f365c03abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:47.279 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.744+0000 7f3679042700 1 --2- 172.21.15.46:0/462076723 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f365c038720 0x7f365c03abe0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3668006fd0 tx=0x7f3668006e40 comp rx=0 tx=0).stop 2024-04-27T03:54:47.279 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.744+0000 7f3679042700 1 -- 172.21.15.46:0/462076723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36741065c0 msgr2=0x7f367419a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:47.279 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.744+0000 7f3679042700 1 --2- 172.21.15.46:0/462076723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36741065c0 0x7f367419a2f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3664011710 tx=0x7f36640047c0 comp rx=0 tx=0).stop 2024-04-27T03:54:47.279 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.744+0000 7f3679042700 1 -- 172.21.15.46:0/462076723 shutdown_connections 2024-04-27T03:54:47.279 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.744+0000 7f3679042700 1 --2- 172.21.15.46:0/462076723 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f365c038720 0x7f365c03abe0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:47.280 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.744+0000 7f3679042700 1 --2- 172.21.15.46:0/462076723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36741065c0 0x7f367419a2f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:47.280 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.745+0000 7f3679042700 1 -- 172.21.15.46:0/462076723 >> 172.21.15.46:0/462076723 conn(0x7f3674101be0 msgr2=0x7f3674102740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:47.280 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.745+0000 7f3679042700 1 -- 172.21.15.46:0/462076723 shutdown_connections 2024-04-27T03:54:47.280 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:46.745+0000 7f3679042700 1 -- 172.21.15.46:0/462076723 wait complete. 2024-04-27T03:54:47.280 INFO:teuthology.orchestra.run.smithi046.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-27T03:54:47.280 INFO:teuthology.orchestra.run.smithi046.stdout:Adding key to root@localhost authorized_keys... 2024-04-27T03:54:47.280 INFO:teuthology.orchestra.run.smithi046.stdout:Adding host smithi046... 2024-04-27T03:54:47.894 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:47 smithi046 ceph-mon[26899]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:54:48.786 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:48 smithi046 ceph-mon[26899]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi046", "addr": "172.21.15.46", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:54:48.786 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:48 smithi046 ceph-mon[26899]: Deploying cephadm binary to smithi046 2024-04-27T03:54:50.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Added host 'smithi046' with addr '172.21.15.46' 2024-04-27T03:54:50.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.674+0000 7fafdd564700 1 Processor -- start 2024-04-27T03:54:50.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.674+0000 7fafdd564700 1 -- start start 2024-04-27T03:54:50.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.674+0000 7fafdd564700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd81070c0 0x7fafd81094b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:50.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.674+0000 7fafdd564700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd8079e50 con 0x7fafd81070c0 2024-04-27T03:54:50.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.674+0000 7fafd6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd81070c0 0x7fafd81094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:50.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.674+0000 7fafd6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd81070c0 0x7fafd81094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34646/0 (socket says 172.21.15.46:34646) 2024-04-27T03:54:50.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.674+0000 7fafd6ffd700 1 -- 172.21.15.46:0/588948109 learned_addr learned my addr 172.21.15.46:0/588948109 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:50.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.674+0000 7fafd6ffd700 1 -- 172.21.15.46:0/588948109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd81099f0 con 0x7fafd81070c0 2024-04-27T03:54:50.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.675+0000 7fafd6ffd700 1 --2- 172.21.15.46:0/588948109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd81070c0 0x7fafd81094b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fafc80098a0 tx=0x7fafc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d22f9bbd1e9d12 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:50.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.675+0000 7fafd5ffb700 1 -- 172.21.15.46:0/588948109 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafc8004180 con 0x7fafd81070c0 2024-04-27T03:54:50.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.675+0000 7fafd5ffb700 1 -- 172.21.15.46:0/588948109 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fafc80042e0 con 0x7fafd81070c0 2024-04-27T03:54:50.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.675+0000 7fafd5ffb700 1 -- 172.21.15.46:0/588948109 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafc8011a70 con 0x7fafd81070c0 2024-04-27T03:54:50.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.675+0000 7fafdd564700 1 -- 172.21.15.46:0/588948109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd81070c0 msgr2=0x7fafd81094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:50.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.675+0000 7fafdd564700 1 --2- 172.21.15.46:0/588948109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd81070c0 0x7fafd81094b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fafc80098a0 tx=0x7fafc8009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:50.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.675+0000 7fafdd564700 1 -- 172.21.15.46:0/588948109 shutdown_connections 2024-04-27T03:54:50.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.675+0000 7fafdd564700 1 --2- 172.21.15.46:0/588948109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd81070c0 0x7fafd81094b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:50.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.675+0000 7fafdd564700 1 -- 172.21.15.46:0/588948109 >> 172.21.15.46:0/588948109 conn(0x7fafd8100af0 msgr2=0x7fafd8102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:50.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.675+0000 7fafdd564700 1 -- 172.21.15.46:0/588948109 shutdown_connections 2024-04-27T03:54:50.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.675+0000 7fafdd564700 1 -- 172.21.15.46:0/588948109 wait complete. 2024-04-27T03:54:50.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.676+0000 7fafdd564700 1 Processor -- start 2024-04-27T03:54:50.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.676+0000 7fafdd564700 1 -- start start 2024-04-27T03:54:50.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.676+0000 7fafdd564700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd81070c0 0x7fafd819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:50.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.676+0000 7fafdd564700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd819ca50 con 0x7fafd81070c0 2024-04-27T03:54:50.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.677+0000 7fafd6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd81070c0 0x7fafd819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:50.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.677+0000 7fafd6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd81070c0 0x7fafd819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34652/0 (socket says 172.21.15.46:34652) 2024-04-27T03:54:50.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.677+0000 7fafd6ffd700 1 -- 172.21.15.46:0/2316331791 learned_addr learned my addr 172.21.15.46:0/2316331791 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:50.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.677+0000 7fafd6ffd700 1 -- 172.21.15.46:0/2316331791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc8009550 con 0x7fafd81070c0 2024-04-27T03:54:50.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.677+0000 7fafd6ffd700 1 --2- 172.21.15.46:0/2316331791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd81070c0 0x7fafd819c510 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fafc8011710 tx=0x7fafc80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:50.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.677+0000 7fafbffff700 1 -- 172.21.15.46:0/2316331791 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafc8003b10 con 0x7fafd81070c0 2024-04-27T03:54:50.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.677+0000 7fafbffff700 1 -- 172.21.15.46:0/2316331791 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fafc801b8c0 con 0x7fafd81070c0 2024-04-27T03:54:50.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.677+0000 7fafdd564700 1 -- 172.21.15.46:0/2316331791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd819cc50 con 0x7fafd81070c0 2024-04-27T03:54:50.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.677+0000 7fafbffff700 1 -- 172.21.15.46:0/2316331791 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafc80197e0 con 0x7fafd81070c0 2024-04-27T03:54:50.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.677+0000 7fafdd564700 1 -- 172.21.15.46:0/2316331791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd819d0f0 con 0x7fafd81070c0 2024-04-27T03:54:50.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.678+0000 7fafbffff700 1 -- 172.21.15.46:0/2316331791 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fafc80199d0 con 0x7fafd81070c0 2024-04-27T03:54:50.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.678+0000 7fafdd564700 1 -- 172.21.15.46:0/2316331791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd810a710 con 0x7fafd81070c0 2024-04-27T03:54:50.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.678+0000 7fafbffff700 1 --2- 172.21.15.46:0/2316331791 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fafc0038770 0x7fafc003ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:50.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.678+0000 7fafbffff700 1 -- 172.21.15.46:0/2316331791 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fafc804ca70 con 0x7fafd81070c0 2024-04-27T03:54:50.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.679+0000 7fafd67fc700 1 --2- 172.21.15.46:0/2316331791 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fafc0038770 0x7fafc003ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:50.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.679+0000 7fafd67fc700 1 --2- 172.21.15.46:0/2316331791 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fafc0038770 0x7fafc003ac30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fafcc006fd0 tx=0x7fafcc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:50.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.683+0000 7fafbffff700 1 -- 172.21.15.46:0/2316331791 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fafc801a750 con 0x7fafd81070c0 2024-04-27T03:54:50.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:47.891+0000 7fafdd564700 1 -- 172.21.15.46:0/2316331791 --> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi046", "addr": "172.21.15.46", "target": ["mon-mgr", ""]}) v1 -- 0x7fafd8101c90 con 0x7fafc0038770 2024-04-27T03:54:50.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:50.396+0000 7fafbffff700 1 -- 172.21.15.46:0/2316331791 <== mgr.14118 v2:172.21.15.46:6800/333567692 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fafd8101c90 con 0x7fafc0038770 2024-04-27T03:54:50.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:50.398+0000 7fafdd564700 1 -- 172.21.15.46:0/2316331791 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fafc0038770 msgr2=0x7fafc003ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:50.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:50.398+0000 7fafdd564700 1 --2- 172.21.15.46:0/2316331791 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fafc0038770 0x7fafc003ac30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fafcc006fd0 tx=0x7fafcc006e40 comp rx=0 tx=0).stop 2024-04-27T03:54:50.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:50.398+0000 7fafdd564700 1 -- 172.21.15.46:0/2316331791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd81070c0 msgr2=0x7fafd819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:50.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:50.398+0000 7fafdd564700 1 --2- 172.21.15.46:0/2316331791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd81070c0 0x7fafd819c510 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fafc8011710 tx=0x7fafc80047c0 comp rx=0 tx=0).stop 2024-04-27T03:54:50.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:50.398+0000 7fafdd564700 1 -- 172.21.15.46:0/2316331791 shutdown_connections 2024-04-27T03:54:50.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:50.398+0000 7fafdd564700 1 --2- 172.21.15.46:0/2316331791 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fafc0038770 0x7fafc003ac30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:50.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:50.398+0000 7fafdd564700 1 --2- 172.21.15.46:0/2316331791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd81070c0 0x7fafd819c510 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:50.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:50.398+0000 7fafdd564700 1 -- 172.21.15.46:0/2316331791 >> 172.21.15.46:0/2316331791 conn(0x7fafd8100af0 msgr2=0x7fafd8101580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:50.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:50.399+0000 7fafdd564700 1 -- 172.21.15.46:0/2316331791 shutdown_connections 2024-04-27T03:54:50.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:50.399+0000 7fafdd564700 1 -- 172.21.15.46:0/2316331791 wait complete. 2024-04-27T03:54:50.953 INFO:teuthology.orchestra.run.smithi046.stdout:Deploying mon service with default placement... 2024-04-27T03:54:51.589 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:51 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:51.589 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:51 smithi046 ceph-mon[26899]: Added host smithi046 2024-04-27T03:54:51.589 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:51 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:54:52.904 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:52 smithi046 ceph-mon[26899]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:54:52.904 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:52 smithi046 ceph-mon[26899]: Saving service mon spec with placement count:5 2024-04-27T03:54:52.904 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:52 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:52.904 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:52 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:52.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-27T03:54:52.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.394+0000 7f853a304700 1 Processor -- start 2024-04-27T03:54:52.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.394+0000 7f853a304700 1 -- start start 2024-04-27T03:54:52.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.395+0000 7f853a304700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85341054e0 0x7f8534105900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:52.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.395+0000 7f853a304700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8534105ed0 con 0x7f85341054e0 2024-04-27T03:54:52.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.395+0000 7f8533fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85341054e0 0x7f8534105900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:52.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.395+0000 7f8533fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85341054e0 0x7f8534105900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38228/0 (socket says 172.21.15.46:38228) 2024-04-27T03:54:52.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.395+0000 7f8533fff700 1 -- 172.21.15.46:0/2467494846 learned_addr learned my addr 172.21.15.46:0/2467494846 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:52.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.395+0000 7f8533fff700 1 -- 172.21.15.46:0/2467494846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85341066e0 con 0x7f85341054e0 2024-04-27T03:54:52.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.396+0000 7f8533fff700 1 --2- 172.21.15.46:0/2467494846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85341054e0 0x7f8534105900 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f85240098a0 tx=0x7f8524009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce6d842bb658e23 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:52.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.397+0000 7f8532ffd700 1 -- 172.21.15.46:0/2467494846 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8524004180 con 0x7f85341054e0 2024-04-27T03:54:52.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.397+0000 7f8532ffd700 1 -- 172.21.15.46:0/2467494846 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f85240042e0 con 0x7f85341054e0 2024-04-27T03:54:52.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.397+0000 7f8532ffd700 1 -- 172.21.15.46:0/2467494846 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8524011a70 con 0x7f85341054e0 2024-04-27T03:54:52.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.398+0000 7f853a304700 1 -- 172.21.15.46:0/2467494846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85341054e0 msgr2=0x7f8534105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:52.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.398+0000 7f853a304700 1 --2- 172.21.15.46:0/2467494846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85341054e0 0x7f8534105900 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f85240098a0 tx=0x7f8524009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:52.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.398+0000 7f853a304700 1 -- 172.21.15.46:0/2467494846 shutdown_connections 2024-04-27T03:54:52.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.398+0000 7f853a304700 1 --2- 172.21.15.46:0/2467494846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85341054e0 0x7f8534105900 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:52.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.398+0000 7f853a304700 1 -- 172.21.15.46:0/2467494846 >> 172.21.15.46:0/2467494846 conn(0x7f8534100ae0 msgr2=0x7f8534102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:52.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.398+0000 7f853a304700 1 -- 172.21.15.46:0/2467494846 shutdown_connections 2024-04-27T03:54:52.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.398+0000 7f853a304700 1 -- 172.21.15.46:0/2467494846 wait complete. 2024-04-27T03:54:52.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.399+0000 7f853a304700 1 Processor -- start 2024-04-27T03:54:52.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.399+0000 7f853a304700 1 -- start start 2024-04-27T03:54:52.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.399+0000 7f853a304700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85341054e0 0x7f8534079a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:52.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.399+0000 7f853a304700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8534079f60 con 0x7f85341054e0 2024-04-27T03:54:52.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.399+0000 7f8533fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85341054e0 0x7f8534079a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:52.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.399+0000 7f8533fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85341054e0 0x7f8534079a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38238/0 (socket says 172.21.15.46:38238) 2024-04-27T03:54:52.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.399+0000 7f8533fff700 1 -- 172.21.15.46:0/2090456387 learned_addr learned my addr 172.21.15.46:0/2090456387 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:52.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.399+0000 7f8533fff700 1 -- 172.21.15.46:0/2090456387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8524009550 con 0x7f85341054e0 2024-04-27T03:54:52.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.400+0000 7f8533fff700 1 --2- 172.21.15.46:0/2090456387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85341054e0 0x7f8534079a20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8524011710 tx=0x7f85240047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:52.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.400+0000 7f85317fa700 1 -- 172.21.15.46:0/2090456387 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8524003b10 con 0x7f85341054e0 2024-04-27T03:54:52.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.400+0000 7f85317fa700 1 -- 172.21.15.46:0/2090456387 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f852401b8c0 con 0x7f85341054e0 2024-04-27T03:54:52.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.400+0000 7f853a304700 1 -- 172.21.15.46:0/2090456387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853419d050 con 0x7f85341054e0 2024-04-27T03:54:52.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.400+0000 7f85317fa700 1 -- 172.21.15.46:0/2090456387 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8524019770 con 0x7f85341054e0 2024-04-27T03:54:52.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.400+0000 7f853a304700 1 -- 172.21.15.46:0/2090456387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853419d490 con 0x7f85341054e0 2024-04-27T03:54:52.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.401+0000 7f85317fa700 1 -- 172.21.15.46:0/2090456387 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f85240199a0 con 0x7f85341054e0 2024-04-27T03:54:52.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.401+0000 7f853a304700 1 -- 172.21.15.46:0/2090456387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8534062510 con 0x7f85341054e0 2024-04-27T03:54:52.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.401+0000 7f85317fa700 1 --2- 172.21.15.46:0/2090456387 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f851c038770 0x7f851c03ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:52.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.401+0000 7f85317fa700 1 -- 172.21.15.46:0/2090456387 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f852404ca20 con 0x7f85341054e0 2024-04-27T03:54:52.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.401+0000 7f85337fe700 1 --2- 172.21.15.46:0/2090456387 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f851c038770 0x7f851c03ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:52.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.402+0000 7f85337fe700 1 --2- 172.21.15.46:0/2090456387 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f851c038770 0x7f851c03ac30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8528006fd0 tx=0x7f8528006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:52.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.406+0000 7f85317fa700 1 -- 172.21.15.46:0/2090456387 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f8524018530 con 0x7f85341054e0 2024-04-27T03:54:52.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.625+0000 7f853a304700 1 -- 172.21.15.46:0/2090456387 --> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f8534101ce0 con 0x7f851c038770 2024-04-27T03:54:52.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.630+0000 7f85317fa700 1 -- 172.21.15.46:0/2090456387 <== mgr.14118 v2:172.21.15.46:6800/333567692 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8534101ce0 con 0x7f851c038770 2024-04-27T03:54:52.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.632+0000 7f853a304700 1 -- 172.21.15.46:0/2090456387 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f851c038770 msgr2=0x7f851c03ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:52.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.632+0000 7f853a304700 1 --2- 172.21.15.46:0/2090456387 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f851c038770 0x7f851c03ac30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8528006fd0 tx=0x7f8528006e40 comp rx=0 tx=0).stop 2024-04-27T03:54:52.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.632+0000 7f853a304700 1 -- 172.21.15.46:0/2090456387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85341054e0 msgr2=0x7f8534079a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:52.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.632+0000 7f853a304700 1 --2- 172.21.15.46:0/2090456387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85341054e0 0x7f8534079a20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8524011710 tx=0x7f85240047c0 comp rx=0 tx=0).stop 2024-04-27T03:54:52.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.632+0000 7f853a304700 1 -- 172.21.15.46:0/2090456387 shutdown_connections 2024-04-27T03:54:52.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.632+0000 7f853a304700 1 --2- 172.21.15.46:0/2090456387 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f851c038770 0x7f851c03ac30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:52.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.632+0000 7f853a304700 1 --2- 172.21.15.46:0/2090456387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85341054e0 0x7f8534079a20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:52.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.632+0000 7f853a304700 1 -- 172.21.15.46:0/2090456387 >> 172.21.15.46:0/2090456387 conn(0x7f8534100ae0 msgr2=0x7f85341015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:52.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.632+0000 7f853a304700 1 -- 172.21.15.46:0/2090456387 shutdown_connections 2024-04-27T03:54:52.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:51.632+0000 7f853a304700 1 -- 172.21.15.46:0/2090456387 wait complete. 2024-04-27T03:54:52.915 INFO:teuthology.orchestra.run.smithi046.stdout:Deploying mgr service with default placement... 2024-04-27T03:54:54.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:53 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:54.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:53 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:54.888 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-27T03:54:54.888 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.581+0000 7fd2b092e700 1 Processor -- start 2024-04-27T03:54:54.888 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.581+0000 7fd2b092e700 1 -- start start 2024-04-27T03:54:54.888 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.581+0000 7fd2b092e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2ac107080 0x7fd2ac109470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:54.888 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.581+0000 7fd2b092e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac074750 con 0x7fd2ac107080 2024-04-27T03:54:54.888 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.582+0000 7fd2aa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2ac107080 0x7fd2ac109470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:54.888 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.582+0000 7fd2aa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2ac107080 0x7fd2ac109470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38254/0 (socket says 172.21.15.46:38254) 2024-04-27T03:54:54.889 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.582+0000 7fd2aa59c700 1 -- 172.21.15.46:0/925205676 learned_addr learned my addr 172.21.15.46:0/925205676 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:54.889 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.582+0000 7fd2aa59c700 1 -- 172.21.15.46:0/925205676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac1099b0 con 0x7fd2ac107080 2024-04-27T03:54:54.889 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.582+0000 7fd2aa59c700 1 --2- 172.21.15.46:0/925205676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2ac107080 0x7fd2ac109470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd2940098a0 tx=0x7fd294009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=811210638d9df733 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:54.889 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.582+0000 7fd2a959a700 1 -- 172.21.15.46:0/925205676 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd294004180 con 0x7fd2ac107080 2024-04-27T03:54:54.889 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.582+0000 7fd2a959a700 1 -- 172.21.15.46:0/925205676 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd2940042e0 con 0x7fd2ac107080 2024-04-27T03:54:54.889 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.582+0000 7fd2a959a700 1 -- 172.21.15.46:0/925205676 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd294011b10 con 0x7fd2ac107080 2024-04-27T03:54:54.889 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.583+0000 7fd2b092e700 1 -- 172.21.15.46:0/925205676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2ac107080 msgr2=0x7fd2ac109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:54.889 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.583+0000 7fd2b092e700 1 --2- 172.21.15.46:0/925205676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2ac107080 0x7fd2ac109470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd2940098a0 tx=0x7fd294009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:54.889 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.583+0000 7fd2b092e700 1 -- 172.21.15.46:0/925205676 shutdown_connections 2024-04-27T03:54:54.889 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.583+0000 7fd2b092e700 1 --2- 172.21.15.46:0/925205676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2ac107080 0x7fd2ac109470 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:54.890 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.583+0000 7fd2b092e700 1 -- 172.21.15.46:0/925205676 >> 172.21.15.46:0/925205676 conn(0x7fd2ac100a50 msgr2=0x7fd2ac102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:54.890 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.583+0000 7fd2b092e700 1 -- 172.21.15.46:0/925205676 shutdown_connections 2024-04-27T03:54:54.890 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.583+0000 7fd2b092e700 1 -- 172.21.15.46:0/925205676 wait complete. 2024-04-27T03:54:54.890 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.584+0000 7fd2b092e700 1 Processor -- start 2024-04-27T03:54:54.890 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.584+0000 7fd2b092e700 1 -- start start 2024-04-27T03:54:54.890 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.584+0000 7fd2b092e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2ac107080 0x7fd2ac19a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:54.890 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.584+0000 7fd2b092e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac19a860 con 0x7fd2ac107080 2024-04-27T03:54:54.890 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.584+0000 7fd2aa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2ac107080 0x7fd2ac19a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:54.891 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.584+0000 7fd2aa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2ac107080 0x7fd2ac19a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38264/0 (socket says 172.21.15.46:38264) 2024-04-27T03:54:54.891 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.584+0000 7fd2aa59c700 1 -- 172.21.15.46:0/2957131347 learned_addr learned my addr 172.21.15.46:0/2957131347 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:54.891 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.585+0000 7fd2aa59c700 1 -- 172.21.15.46:0/2957131347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd294009550 con 0x7fd2ac107080 2024-04-27T03:54:54.891 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.585+0000 7fd2aa59c700 1 --2- 172.21.15.46:0/2957131347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2ac107080 0x7fd2ac19a320 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd294009fb0 tx=0x7fd294011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:54.891 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.585+0000 7fd2a37fe700 1 -- 172.21.15.46:0/2957131347 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2940038b0 con 0x7fd2ac107080 2024-04-27T03:54:54.891 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.585+0000 7fd2a37fe700 1 -- 172.21.15.46:0/2957131347 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd294003a10 con 0x7fd2ac107080 2024-04-27T03:54:54.891 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.585+0000 7fd2b092e700 1 -- 172.21.15.46:0/2957131347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ac19aa60 con 0x7fd2ac107080 2024-04-27T03:54:54.891 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.585+0000 7fd2b092e700 1 -- 172.21.15.46:0/2957131347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ac19af00 con 0x7fd2ac107080 2024-04-27T03:54:54.891 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.585+0000 7fd2a37fe700 1 -- 172.21.15.46:0/2957131347 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd294019750 con 0x7fd2ac107080 2024-04-27T03:54:54.891 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.586+0000 7fd2a37fe700 1 -- 172.21.15.46:0/2957131347 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fd2940198b0 con 0x7fd2ac107080 2024-04-27T03:54:54.891 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.586+0000 7fd2b092e700 1 -- 172.21.15.46:0/2957131347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ac10a740 con 0x7fd2ac107080 2024-04-27T03:54:54.891 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.587+0000 7fd2a37fe700 1 --2- 172.21.15.46:0/2957131347 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fd298038730 0x7fd29803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:54.891 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.587+0000 7fd2a37fe700 1 -- 172.21.15.46:0/2957131347 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd29404c7d0 con 0x7fd2ac107080 2024-04-27T03:54:54.892 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.587+0000 7fd2a9d9b700 1 --2- 172.21.15.46:0/2957131347 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fd298038730 0x7fd29803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:54.892 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.587+0000 7fd2a9d9b700 1 --2- 172.21.15.46:0/2957131347 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fd298038730 0x7fd29803abf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd29c006fd0 tx=0x7fd29c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:54.892 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.592+0000 7fd2a37fe700 1 -- 172.21.15.46:0/2957131347 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fd2940229a0 con 0x7fd2ac107080 2024-04-27T03:54:54.892 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.843+0000 7fd2b092e700 1 -- 172.21.15.46:0/2957131347 --> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2ac101d40 con 0x7fd298038730 2024-04-27T03:54:54.892 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.847+0000 7fd2a37fe700 1 -- 172.21.15.46:0/2957131347 <== mgr.14118 v2:172.21.15.46:6800/333567692 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd2ac101d40 con 0x7fd298038730 2024-04-27T03:54:54.892 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.849+0000 7fd2b092e700 1 -- 172.21.15.46:0/2957131347 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fd298038730 msgr2=0x7fd29803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:54.892 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.849+0000 7fd2b092e700 1 --2- 172.21.15.46:0/2957131347 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fd298038730 0x7fd29803abf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd29c006fd0 tx=0x7fd29c006e40 comp rx=0 tx=0).stop 2024-04-27T03:54:54.892 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.849+0000 7fd2b092e700 1 -- 172.21.15.46:0/2957131347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2ac107080 msgr2=0x7fd2ac19a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:54.892 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.849+0000 7fd2b092e700 1 --2- 172.21.15.46:0/2957131347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2ac107080 0x7fd2ac19a320 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd294009fb0 tx=0x7fd294011890 comp rx=0 tx=0).stop 2024-04-27T03:54:54.892 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.849+0000 7fd2b092e700 1 -- 172.21.15.46:0/2957131347 shutdown_connections 2024-04-27T03:54:54.892 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.849+0000 7fd2b092e700 1 --2- 172.21.15.46:0/2957131347 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fd298038730 0x7fd29803abf0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:54.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.849+0000 7fd2b092e700 1 --2- 172.21.15.46:0/2957131347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2ac107080 0x7fd2ac19a320 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:54.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.849+0000 7fd2b092e700 1 -- 172.21.15.46:0/2957131347 >> 172.21.15.46:0/2957131347 conn(0x7fd2ac100a50 msgr2=0x7fd2ac101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:54.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.849+0000 7fd2b092e700 1 -- 172.21.15.46:0/2957131347 shutdown_connections 2024-04-27T03:54:54.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:53.849+0000 7fd2b092e700 1 -- 172.21.15.46:0/2957131347 wait complete. 2024-04-27T03:54:54.893 INFO:teuthology.orchestra.run.smithi046.stdout:Deploying crash service with default placement... 2024-04-27T03:54:55.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:55 smithi046 ceph-mon[26899]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:54:55.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:55 smithi046 ceph-mon[26899]: Saving service mgr spec with placement count:2 2024-04-27T03:54:55.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:55 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:55.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:55 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:56.145 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-27T03:54:56.145 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.344+0000 7ff2a4b91700 1 Processor -- start 2024-04-27T03:54:56.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.344+0000 7ff2a4b91700 1 -- start start 2024-04-27T03:54:56.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.344+0000 7ff2a4b91700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2a0107020 0x7ff2a0109410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:56.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.344+0000 7ff2a4b91700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a00747b0 con 0x7ff2a0107020 2024-04-27T03:54:56.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.344+0000 7ff29e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2a0107020 0x7ff2a0109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:56.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.345+0000 7ff29e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2a0107020 0x7ff2a0109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38280/0 (socket says 172.21.15.46:38280) 2024-04-27T03:54:56.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.345+0000 7ff29e59c700 1 -- 172.21.15.46:0/1977283412 learned_addr learned my addr 172.21.15.46:0/1977283412 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:56.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.345+0000 7ff29e59c700 1 -- 172.21.15.46:0/1977283412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a0109950 con 0x7ff2a0107020 2024-04-27T03:54:56.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.345+0000 7ff29e59c700 1 --2- 172.21.15.46:0/1977283412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2a0107020 0x7ff2a0109410 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff2900098a0 tx=0x7ff290009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86116b082ad05d60 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:56.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.345+0000 7ff29d59a700 1 -- 172.21.15.46:0/1977283412 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff290004180 con 0x7ff2a0107020 2024-04-27T03:54:56.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.345+0000 7ff29d59a700 1 -- 172.21.15.46:0/1977283412 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2900042e0 con 0x7ff2a0107020 2024-04-27T03:54:56.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.345+0000 7ff29d59a700 1 -- 172.21.15.46:0/1977283412 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff290011b10 con 0x7ff2a0107020 2024-04-27T03:54:56.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.346+0000 7ff2a4b91700 1 -- 172.21.15.46:0/1977283412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2a0107020 msgr2=0x7ff2a0109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:56.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.346+0000 7ff2a4b91700 1 --2- 172.21.15.46:0/1977283412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2a0107020 0x7ff2a0109410 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff2900098a0 tx=0x7ff290009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:56.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.346+0000 7ff2a4b91700 1 -- 172.21.15.46:0/1977283412 shutdown_connections 2024-04-27T03:54:56.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.346+0000 7ff2a4b91700 1 --2- 172.21.15.46:0/1977283412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2a0107020 0x7ff2a0109410 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:56.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.346+0000 7ff2a4b91700 1 -- 172.21.15.46:0/1977283412 >> 172.21.15.46:0/1977283412 conn(0x7ff2a0100a50 msgr2=0x7ff2a0102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:56.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.346+0000 7ff2a4b91700 1 -- 172.21.15.46:0/1977283412 shutdown_connections 2024-04-27T03:54:56.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.346+0000 7ff2a4b91700 1 -- 172.21.15.46:0/1977283412 wait complete. 2024-04-27T03:54:56.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.347+0000 7ff2a4b91700 1 Processor -- start 2024-04-27T03:54:56.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.347+0000 7ff2a4b91700 1 -- start start 2024-04-27T03:54:56.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.347+0000 7ff2a4b91700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2a0107020 0x7ff2a0106850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:56.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.347+0000 7ff2a4b91700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a0106d90 con 0x7ff2a0107020 2024-04-27T03:54:56.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.347+0000 7ff29e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2a0107020 0x7ff2a0106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:56.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.347+0000 7ff29e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2a0107020 0x7ff2a0106850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38286/0 (socket says 172.21.15.46:38286) 2024-04-27T03:54:56.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.347+0000 7ff29e59c700 1 -- 172.21.15.46:0/132118996 learned_addr learned my addr 172.21.15.46:0/132118996 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:56.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.347+0000 7ff29e59c700 1 -- 172.21.15.46:0/132118996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff290009550 con 0x7ff2a0107020 2024-04-27T03:54:56.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.347+0000 7ff29e59c700 1 --2- 172.21.15.46:0/132118996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2a0107020 0x7ff2a0106850 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff290009fb0 tx=0x7ff290011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:56.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.348+0000 7ff2977fe700 1 -- 172.21.15.46:0/132118996 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2900038b0 con 0x7ff2a0107020 2024-04-27T03:54:56.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.348+0000 7ff2977fe700 1 -- 172.21.15.46:0/132118996 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff290003a10 con 0x7ff2a0107020 2024-04-27T03:54:56.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.348+0000 7ff2977fe700 1 -- 172.21.15.46:0/132118996 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff290019750 con 0x7ff2a0107020 2024-04-27T03:54:56.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.348+0000 7ff2a4b91700 1 -- 172.21.15.46:0/132118996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a0104fd0 con 0x7ff2a0107020 2024-04-27T03:54:56.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.348+0000 7ff2a4b91700 1 -- 172.21.15.46:0/132118996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a0105470 con 0x7ff2a0107020 2024-04-27T03:54:56.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.349+0000 7ff2977fe700 1 -- 172.21.15.46:0/132118996 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7ff29001ac10 con 0x7ff2a0107020 2024-04-27T03:54:56.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.349+0000 7ff2a4b91700 1 -- 172.21.15.46:0/132118996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a010a740 con 0x7ff2a0107020 2024-04-27T03:54:56.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.349+0000 7ff2977fe700 1 --2- 172.21.15.46:0/132118996 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7ff28c0386e0 0x7ff28c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:56.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.349+0000 7ff2977fe700 1 -- 172.21.15.46:0/132118996 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff29004c830 con 0x7ff2a0107020 2024-04-27T03:54:56.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.349+0000 7ff29dd9b700 1 --2- 172.21.15.46:0/132118996 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7ff28c0386e0 0x7ff28c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:56.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.350+0000 7ff29dd9b700 1 --2- 172.21.15.46:0/132118996 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7ff28c0386e0 0x7ff28c03aba0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff288006fd0 tx=0x7ff288006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:56.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.354+0000 7ff2977fe700 1 -- 172.21.15.46:0/132118996 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7ff290022720 con 0x7ff2a0107020 2024-04-27T03:54:56.149 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.578+0000 7ff2a4b91700 1 -- 172.21.15.46:0/132118996 --> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2a0105c60 con 0x7ff28c0386e0 2024-04-27T03:54:56.149 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.582+0000 7ff2977fe700 1 -- 172.21.15.46:0/132118996 <== mgr.14118 v2:172.21.15.46:6800/333567692 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7ff2a0105c60 con 0x7ff28c0386e0 2024-04-27T03:54:56.149 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.584+0000 7ff2a4b91700 1 -- 172.21.15.46:0/132118996 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7ff28c0386e0 msgr2=0x7ff28c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:56.149 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.584+0000 7ff2a4b91700 1 --2- 172.21.15.46:0/132118996 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7ff28c0386e0 0x7ff28c03aba0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff288006fd0 tx=0x7ff288006e40 comp rx=0 tx=0).stop 2024-04-27T03:54:56.149 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.584+0000 7ff2a4b91700 1 -- 172.21.15.46:0/132118996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2a0107020 msgr2=0x7ff2a0106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:56.149 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.584+0000 7ff2a4b91700 1 --2- 172.21.15.46:0/132118996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2a0107020 0x7ff2a0106850 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff290009fb0 tx=0x7ff290011890 comp rx=0 tx=0).stop 2024-04-27T03:54:56.149 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.584+0000 7ff2a4b91700 1 -- 172.21.15.46:0/132118996 shutdown_connections 2024-04-27T03:54:56.149 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.584+0000 7ff2a4b91700 1 --2- 172.21.15.46:0/132118996 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7ff28c0386e0 0x7ff28c03aba0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:56.149 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.584+0000 7ff2a4b91700 1 --2- 172.21.15.46:0/132118996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2a0107020 0x7ff2a0106850 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:56.149 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.585+0000 7ff2a4b91700 1 -- 172.21.15.46:0/132118996 >> 172.21.15.46:0/132118996 conn(0x7ff2a0100a50 msgr2=0x7ff2a01015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:56.149 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.585+0000 7ff2a4b91700 1 -- 172.21.15.46:0/132118996 shutdown_connections 2024-04-27T03:54:56.149 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:55.585+0000 7ff2a4b91700 1 -- 172.21.15.46:0/132118996 wait complete. 2024-04-27T03:54:56.149 INFO:teuthology.orchestra.run.smithi046.stdout:Deploying ceph-exporter service with default placement... 2024-04-27T03:54:56.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:56 smithi046 ceph-mon[26899]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:54:56.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:56 smithi046 ceph-mon[26899]: Saving service crash spec with placement * 2024-04-27T03:54:56.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:56 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:58.167 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-04-27T03:54:58.167 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.205+0000 7f8140dc7700 1 Processor -- start 2024-04-27T03:54:58.167 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.205+0000 7f8140dc7700 1 -- start start 2024-04-27T03:54:58.167 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.205+0000 7f8140dc7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f813c1065c0 0x7f813c1069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:58.167 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.205+0000 7f8140dc7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c106fb0 con 0x7f813c1065c0 2024-04-27T03:54:58.167 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.206+0000 7f813a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f813c1065c0 0x7f813c1069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:58.167 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.206+0000 7f813a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f813c1065c0 0x7f813c1069e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38294/0 (socket says 172.21.15.46:38294) 2024-04-27T03:54:58.167 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.206+0000 7f813a59c700 1 -- 172.21.15.46:0/3588854957 learned_addr learned my addr 172.21.15.46:0/3588854957 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:58.168 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.206+0000 7f813a59c700 1 -- 172.21.15.46:0/3588854957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c1077c0 con 0x7f813c1065c0 2024-04-27T03:54:58.168 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.206+0000 7f813a59c700 1 --2- 172.21.15.46:0/3588854957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f813c1065c0 0x7f813c1069e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f812c0098a0 tx=0x7f812c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a076760500ba657 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:58.168 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.206+0000 7f813959a700 1 -- 172.21.15.46:0/3588854957 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f812c004180 con 0x7f813c1065c0 2024-04-27T03:54:58.168 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.206+0000 7f813959a700 1 -- 172.21.15.46:0/3588854957 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f812c0042e0 con 0x7f813c1065c0 2024-04-27T03:54:58.168 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.206+0000 7f813959a700 1 -- 172.21.15.46:0/3588854957 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f812c011b10 con 0x7f813c1065c0 2024-04-27T03:54:58.168 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.207+0000 7f8140dc7700 1 -- 172.21.15.46:0/3588854957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f813c1065c0 msgr2=0x7f813c1069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:58.168 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.207+0000 7f8140dc7700 1 --2- 172.21.15.46:0/3588854957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f813c1065c0 0x7f813c1069e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f812c0098a0 tx=0x7f812c009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:58.168 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.207+0000 7f8140dc7700 1 -- 172.21.15.46:0/3588854957 shutdown_connections 2024-04-27T03:54:58.168 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.207+0000 7f8140dc7700 1 --2- 172.21.15.46:0/3588854957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f813c1065c0 0x7f813c1069e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:58.168 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.207+0000 7f8140dc7700 1 -- 172.21.15.46:0/3588854957 >> 172.21.15.46:0/3588854957 conn(0x7f813c101be0 msgr2=0x7f813c104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:58.168 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.207+0000 7f8140dc7700 1 -- 172.21.15.46:0/3588854957 shutdown_connections 2024-04-27T03:54:58.168 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.207+0000 7f8140dc7700 1 -- 172.21.15.46:0/3588854957 wait complete. 2024-04-27T03:54:58.168 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.208+0000 7f8140dc7700 1 Processor -- start 2024-04-27T03:54:58.168 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.208+0000 7f8140dc7700 1 -- start start 2024-04-27T03:54:58.169 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.208+0000 7f8140dc7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f813c1065c0 0x7f813c19a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:58.169 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.208+0000 7f8140dc7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c19a870 con 0x7f813c1065c0 2024-04-27T03:54:58.169 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.208+0000 7f813a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f813c1065c0 0x7f813c19a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:58.169 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.209+0000 7f813a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f813c1065c0 0x7f813c19a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38304/0 (socket says 172.21.15.46:38304) 2024-04-27T03:54:58.169 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.209+0000 7f813a59c700 1 -- 172.21.15.46:0/3830184362 learned_addr learned my addr 172.21.15.46:0/3830184362 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:58.169 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.209+0000 7f813a59c700 1 -- 172.21.15.46:0/3830184362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c009550 con 0x7f813c1065c0 2024-04-27T03:54:58.169 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.209+0000 7f813a59c700 1 --2- 172.21.15.46:0/3830184362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f813c1065c0 0x7f813c19a330 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f812c0117b0 tx=0x7f812c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:58.169 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.209+0000 7f812b7fe700 1 -- 172.21.15.46:0/3830184362 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f812c0038b0 con 0x7f813c1065c0 2024-04-27T03:54:58.169 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.209+0000 7f812b7fe700 1 -- 172.21.15.46:0/3830184362 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f812c003a10 con 0x7f813c1065c0 2024-04-27T03:54:58.169 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.209+0000 7f8140dc7700 1 -- 172.21.15.46:0/3830184362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813c19aa70 con 0x7f813c1065c0 2024-04-27T03:54:58.169 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.209+0000 7f812b7fe700 1 -- 172.21.15.46:0/3830184362 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f812c0246e0 con 0x7f813c1065c0 2024-04-27T03:54:58.169 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.209+0000 7f8140dc7700 1 -- 172.21.15.46:0/3830184362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813c19af10 con 0x7f813c1065c0 2024-04-27T03:54:58.169 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.210+0000 7f812b7fe700 1 -- 172.21.15.46:0/3830184362 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f812c02d3f0 con 0x7f813c1065c0 2024-04-27T03:54:58.170 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.210+0000 7f812b7fe700 1 --2- 172.21.15.46:0/3830184362 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f8124038770 0x7f812403ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:58.170 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.210+0000 7f812b7fe700 1 -- 172.21.15.46:0/3830184362 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f812c04c800 con 0x7f813c1065c0 2024-04-27T03:54:58.170 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.210+0000 7f8140dc7700 1 -- 172.21.15.46:0/3830184362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813c062510 con 0x7f813c1065c0 2024-04-27T03:54:58.170 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.210+0000 7f8139d9b700 1 --2- 172.21.15.46:0/3830184362 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f8124038770 0x7f812403ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:58.170 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.211+0000 7f8139d9b700 1 --2- 172.21.15.46:0/3830184362 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f8124038770 0x7f812403ac30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8130006fd0 tx=0x7f8130006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:58.170 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.215+0000 7f812b7fe700 1 -- 172.21.15.46:0/3830184362 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f812c02d6f0 con 0x7f813c1065c0 2024-04-27T03:54:58.170 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.434+0000 7f8140dc7700 1 -- 172.21.15.46:0/3830184362 --> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f813c19d7a0 con 0x7f8124038770 2024-04-27T03:54:58.170 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.439+0000 7f812b7fe700 1 -- 172.21.15.46:0/3830184362 <== mgr.14118 v2:172.21.15.46:6800/333567692 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f813c19d7a0 con 0x7f8124038770 2024-04-27T03:54:58.170 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.441+0000 7f8140dc7700 1 -- 172.21.15.46:0/3830184362 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f8124038770 msgr2=0x7f812403ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:58.170 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.441+0000 7f8140dc7700 1 --2- 172.21.15.46:0/3830184362 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f8124038770 0x7f812403ac30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8130006fd0 tx=0x7f8130006e40 comp rx=0 tx=0).stop 2024-04-27T03:54:58.170 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.441+0000 7f8140dc7700 1 -- 172.21.15.46:0/3830184362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f813c1065c0 msgr2=0x7f813c19a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:58.170 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.441+0000 7f8140dc7700 1 --2- 172.21.15.46:0/3830184362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f813c1065c0 0x7f813c19a330 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f812c0117b0 tx=0x7f812c011890 comp rx=0 tx=0).stop 2024-04-27T03:54:58.170 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.441+0000 7f8140dc7700 1 -- 172.21.15.46:0/3830184362 shutdown_connections 2024-04-27T03:54:58.171 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.441+0000 7f8140dc7700 1 --2- 172.21.15.46:0/3830184362 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f8124038770 0x7f812403ac30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:58.171 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.441+0000 7f8140dc7700 1 --2- 172.21.15.46:0/3830184362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f813c1065c0 0x7f813c19a330 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:58.171 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.441+0000 7f8140dc7700 1 -- 172.21.15.46:0/3830184362 >> 172.21.15.46:0/3830184362 conn(0x7f813c101be0 msgr2=0x7f813c102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:58.171 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.442+0000 7f8140dc7700 1 -- 172.21.15.46:0/3830184362 shutdown_connections 2024-04-27T03:54:58.171 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:57.442+0000 7f8140dc7700 1 -- 172.21.15.46:0/3830184362 wait complete. 2024-04-27T03:54:58.171 INFO:teuthology.orchestra.run.smithi046.stdout:Deploying prometheus service with default placement... 2024-04-27T03:54:58.561 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:58 smithi046 ceph-mon[26899]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:54:58.561 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:58 smithi046 ceph-mon[26899]: Saving service ceph-exporter spec with placement * 2024-04-27T03:54:58.561 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:58 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:54:59.339 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-27T03:54:59.339 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.585+0000 7fa884a44700 1 Processor -- start 2024-04-27T03:54:59.340 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.585+0000 7fa884a44700 1 -- start start 2024-04-27T03:54:59.340 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.585+0000 7fa884a44700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa880108800 0x7fa880108c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:59.340 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.585+0000 7fa884a44700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8801091f0 con 0x7fa880108800 2024-04-27T03:54:59.340 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.585+0000 7fa87e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa880108800 0x7fa880108c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:59.340 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.585+0000 7fa87e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa880108800 0x7fa880108c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38318/0 (socket says 172.21.15.46:38318) 2024-04-27T03:54:59.340 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.585+0000 7fa87e59c700 1 -- 172.21.15.46:0/558751301 learned_addr learned my addr 172.21.15.46:0/558751301 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:59.340 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.586+0000 7fa87e59c700 1 -- 172.21.15.46:0/558751301 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8801099b0 con 0x7fa880108800 2024-04-27T03:54:59.340 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.586+0000 7fa87e59c700 1 --2- 172.21.15.46:0/558751301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa880108800 0x7fa880108c20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa8700098a0 tx=0x7fa870009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c690a6a56d5b78 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:59.340 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.586+0000 7fa87d59a700 1 -- 172.21.15.46:0/558751301 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa870004180 con 0x7fa880108800 2024-04-27T03:54:59.340 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.586+0000 7fa87d59a700 1 -- 172.21.15.46:0/558751301 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8700042e0 con 0x7fa880108800 2024-04-27T03:54:59.340 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.586+0000 7fa87d59a700 1 -- 172.21.15.46:0/558751301 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa870011b10 con 0x7fa880108800 2024-04-27T03:54:59.340 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.586+0000 7fa884a44700 1 -- 172.21.15.46:0/558751301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa880108800 msgr2=0x7fa880108c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:59.340 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.586+0000 7fa884a44700 1 --2- 172.21.15.46:0/558751301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa880108800 0x7fa880108c20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa8700098a0 tx=0x7fa870009bb0 comp rx=0 tx=0).stop 2024-04-27T03:54:59.341 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.587+0000 7fa884a44700 1 -- 172.21.15.46:0/558751301 shutdown_connections 2024-04-27T03:54:59.341 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.587+0000 7fa884a44700 1 --2- 172.21.15.46:0/558751301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa880108800 0x7fa880108c20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:59.341 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.587+0000 7fa884a44700 1 -- 172.21.15.46:0/558751301 >> 172.21.15.46:0/558751301 conn(0x7fa88007bd90 msgr2=0x7fa8801064f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:59.341 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.587+0000 7fa884a44700 1 -- 172.21.15.46:0/558751301 shutdown_connections 2024-04-27T03:54:59.341 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.587+0000 7fa884a44700 1 -- 172.21.15.46:0/558751301 wait complete. 2024-04-27T03:54:59.341 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.587+0000 7fa884a44700 1 Processor -- start 2024-04-27T03:54:59.341 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.587+0000 7fa884a44700 1 -- start start 2024-04-27T03:54:59.341 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.588+0000 7fa884a44700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa880108800 0x7fa88019c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:59.341 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.588+0000 7fa884a44700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88019caa0 con 0x7fa880108800 2024-04-27T03:54:59.341 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.588+0000 7fa87e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa880108800 0x7fa88019c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:59.341 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.588+0000 7fa87e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa880108800 0x7fa88019c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38330/0 (socket says 172.21.15.46:38330) 2024-04-27T03:54:59.341 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.588+0000 7fa87e59c700 1 -- 172.21.15.46:0/2760304753 learned_addr learned my addr 172.21.15.46:0/2760304753 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:54:59.341 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.588+0000 7fa87e59c700 1 -- 172.21.15.46:0/2760304753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa870009550 con 0x7fa880108800 2024-04-27T03:54:59.342 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.588+0000 7fa87e59c700 1 --2- 172.21.15.46:0/2760304753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa880108800 0x7fa88019c560 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa8700117b0 tx=0x7fa870011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:59.342 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.588+0000 7fa86f7fe700 1 -- 172.21.15.46:0/2760304753 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8700038b0 con 0x7fa880108800 2024-04-27T03:54:59.342 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.588+0000 7fa86f7fe700 1 -- 172.21.15.46:0/2760304753 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa870003a10 con 0x7fa880108800 2024-04-27T03:54:59.342 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.588+0000 7fa884a44700 1 -- 172.21.15.46:0/2760304753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa88019cca0 con 0x7fa880108800 2024-04-27T03:54:59.342 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.588+0000 7fa86f7fe700 1 -- 172.21.15.46:0/2760304753 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8700246e0 con 0x7fa880108800 2024-04-27T03:54:59.342 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.589+0000 7fa884a44700 1 -- 172.21.15.46:0/2760304753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa88019d140 con 0x7fa880108800 2024-04-27T03:54:59.342 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.589+0000 7fa86f7fe700 1 -- 172.21.15.46:0/2760304753 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fa87002d3f0 con 0x7fa880108800 2024-04-27T03:54:59.342 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.589+0000 7fa884a44700 1 -- 172.21.15.46:0/2760304753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa88010cbe0 con 0x7fa880108800 2024-04-27T03:54:59.342 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.590+0000 7fa86f7fe700 1 --2- 172.21.15.46:0/2760304753 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa868038720 0x7fa86803abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:54:59.342 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.590+0000 7fa86f7fe700 1 -- 172.21.15.46:0/2760304753 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa87004caa0 con 0x7fa880108800 2024-04-27T03:54:59.342 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.590+0000 7fa87dd9b700 1 --2- 172.21.15.46:0/2760304753 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa868038720 0x7fa86803abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:54:59.342 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.590+0000 7fa87dd9b700 1 --2- 172.21.15.46:0/2760304753 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa868038720 0x7fa86803abe0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa874006fd0 tx=0x7fa874006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:54:59.342 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.595+0000 7fa86f7fe700 1 -- 172.21.15.46:0/2760304753 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fa870010970 con 0x7fa880108800 2024-04-27T03:54:59.343 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.810+0000 7fa884a44700 1 -- 172.21.15.46:0/2760304753 --> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fa88019f980 con 0x7fa868038720 2024-04-27T03:54:59.343 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.815+0000 7fa86f7fe700 1 -- 172.21.15.46:0/2760304753 <== mgr.14118 v2:172.21.15.46:6800/333567692 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fa88019f980 con 0x7fa868038720 2024-04-27T03:54:59.343 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.816+0000 7fa884a44700 1 -- 172.21.15.46:0/2760304753 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa868038720 msgr2=0x7fa86803abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:59.343 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.816+0000 7fa884a44700 1 --2- 172.21.15.46:0/2760304753 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa868038720 0x7fa86803abe0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa874006fd0 tx=0x7fa874006e40 comp rx=0 tx=0).stop 2024-04-27T03:54:59.343 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.817+0000 7fa884a44700 1 -- 172.21.15.46:0/2760304753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa880108800 msgr2=0x7fa88019c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:54:59.343 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.817+0000 7fa884a44700 1 --2- 172.21.15.46:0/2760304753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa880108800 0x7fa88019c560 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa8700117b0 tx=0x7fa870011890 comp rx=0 tx=0).stop 2024-04-27T03:54:59.343 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.817+0000 7fa884a44700 1 -- 172.21.15.46:0/2760304753 shutdown_connections 2024-04-27T03:54:59.343 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.817+0000 7fa884a44700 1 --2- 172.21.15.46:0/2760304753 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa868038720 0x7fa86803abe0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:59.343 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.817+0000 7fa884a44700 1 --2- 172.21.15.46:0/2760304753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa880108800 0x7fa88019c560 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:54:59.343 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.817+0000 7fa884a44700 1 -- 172.21.15.46:0/2760304753 >> 172.21.15.46:0/2760304753 conn(0x7fa88007bd90 msgr2=0x7fa880105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:54:59.343 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.817+0000 7fa884a44700 1 -- 172.21.15.46:0/2760304753 shutdown_connections 2024-04-27T03:54:59.343 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:58.817+0000 7fa884a44700 1 -- 172.21.15.46:0/2760304753 wait complete. 2024-04-27T03:54:59.343 INFO:teuthology.orchestra.run.smithi046.stdout:Deploying grafana service with default placement... 2024-04-27T03:54:59.977 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:59 smithi046 ceph-mon[26899]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:54:59.977 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:59 smithi046 ceph-mon[26899]: Saving service prometheus spec with placement count:1 2024-04-27T03:54:59.977 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:54:59 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:00.563 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-27T03:55:00.563 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.744+0000 7f68c9b49700 1 Processor -- start 2024-04-27T03:55:00.563 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.744+0000 7f68c9b49700 1 -- start start 2024-04-27T03:55:00.563 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.744+0000 7f68c9b49700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c41065e0 0x7f68c4106a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:00.563 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.744+0000 7f68c9b49700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c4106fd0 con 0x7f68c41065e0 2024-04-27T03:55:00.563 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.745+0000 7f68c37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c41065e0 0x7f68c4106a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:00.563 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.745+0000 7f68c37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c41065e0 0x7f68c4106a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38336/0 (socket says 172.21.15.46:38336) 2024-04-27T03:55:00.563 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.745+0000 7f68c37fe700 1 -- 172.21.15.46:0/675843176 learned_addr learned my addr 172.21.15.46:0/675843176 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:00.564 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.745+0000 7f68c37fe700 1 -- 172.21.15.46:0/675843176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c4107790 con 0x7f68c41065e0 2024-04-27T03:55:00.564 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.745+0000 7f68c37fe700 1 --2- 172.21.15.46:0/675843176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c41065e0 0x7f68c4106a00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f68b80098a0 tx=0x7f68b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582dd0f1ef2b9d00 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:00.564 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.745+0000 7f68c27fc700 1 -- 172.21.15.46:0/675843176 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68b8004180 con 0x7f68c41065e0 2024-04-27T03:55:00.564 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.745+0000 7f68c27fc700 1 -- 172.21.15.46:0/675843176 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68b80042e0 con 0x7f68c41065e0 2024-04-27T03:55:00.564 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.745+0000 7f68c27fc700 1 -- 172.21.15.46:0/675843176 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68b8011b10 con 0x7f68c41065e0 2024-04-27T03:55:00.564 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.746+0000 7f68c9b49700 1 -- 172.21.15.46:0/675843176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c41065e0 msgr2=0x7f68c4106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:00.564 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.746+0000 7f68c9b49700 1 --2- 172.21.15.46:0/675843176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c41065e0 0x7f68c4106a00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f68b80098a0 tx=0x7f68b8009bb0 comp rx=0 tx=0).stop 2024-04-27T03:55:00.564 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.746+0000 7f68c9b49700 1 -- 172.21.15.46:0/675843176 shutdown_connections 2024-04-27T03:55:00.564 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.746+0000 7f68c9b49700 1 --2- 172.21.15.46:0/675843176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c41065e0 0x7f68c4106a00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:00.564 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.746+0000 7f68c9b49700 1 -- 172.21.15.46:0/675843176 >> 172.21.15.46:0/675843176 conn(0x7f68c4101b80 msgr2=0x7f68c4103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:00.564 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.746+0000 7f68c9b49700 1 -- 172.21.15.46:0/675843176 shutdown_connections 2024-04-27T03:55:00.564 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.746+0000 7f68c9b49700 1 -- 172.21.15.46:0/675843176 wait complete. 2024-04-27T03:55:00.564 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.747+0000 7f68c9b49700 1 Processor -- start 2024-04-27T03:55:00.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.747+0000 7f68c9b49700 1 -- start start 2024-04-27T03:55:00.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.747+0000 7f68c9b49700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c41065e0 0x7f68c419a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:00.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.747+0000 7f68c9b49700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c419a880 con 0x7f68c41065e0 2024-04-27T03:55:00.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.748+0000 7f68c37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c41065e0 0x7f68c419a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:00.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.748+0000 7f68c37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c41065e0 0x7f68c419a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38340/0 (socket says 172.21.15.46:38340) 2024-04-27T03:55:00.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.748+0000 7f68c37fe700 1 -- 172.21.15.46:0/2883308492 learned_addr learned my addr 172.21.15.46:0/2883308492 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:00.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.748+0000 7f68c37fe700 1 -- 172.21.15.46:0/2883308492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b8009550 con 0x7f68c41065e0 2024-04-27T03:55:00.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.748+0000 7f68c37fe700 1 --2- 172.21.15.46:0/2883308492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c41065e0 0x7f68c419a340 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f68b8009fb0 tx=0x7f68b8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:00.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.748+0000 7f68c0ff9700 1 -- 172.21.15.46:0/2883308492 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68b80038b0 con 0x7f68c41065e0 2024-04-27T03:55:00.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.748+0000 7f68c0ff9700 1 -- 172.21.15.46:0/2883308492 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68b8003a10 con 0x7f68c41065e0 2024-04-27T03:55:00.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.748+0000 7f68c9b49700 1 -- 172.21.15.46:0/2883308492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c419aa80 con 0x7f68c41065e0 2024-04-27T03:55:00.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.748+0000 7f68c0ff9700 1 -- 172.21.15.46:0/2883308492 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68b8019770 con 0x7f68c41065e0 2024-04-27T03:55:00.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.748+0000 7f68c9b49700 1 -- 172.21.15.46:0/2883308492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c419af20 con 0x7f68c41065e0 2024-04-27T03:55:00.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.749+0000 7f68c0ff9700 1 -- 172.21.15.46:0/2883308492 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f68b80198d0 con 0x7f68c41065e0 2024-04-27T03:55:00.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.749+0000 7f68c0ff9700 1 --2- 172.21.15.46:0/2883308492 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f68ac0386e0 0x7f68ac03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:00.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.749+0000 7f68c0ff9700 1 -- 172.21.15.46:0/2883308492 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f68b804c7d0 con 0x7f68c41065e0 2024-04-27T03:55:00.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.750+0000 7f68c2ffd700 1 --2- 172.21.15.46:0/2883308492 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f68ac0386e0 0x7f68ac03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:00.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.750+0000 7f68c9b49700 1 -- 172.21.15.46:0/2883308492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c4062510 con 0x7f68c41065e0 2024-04-27T03:55:00.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.750+0000 7f68c2ffd700 1 --2- 172.21.15.46:0/2883308492 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f68ac0386e0 0x7f68ac03aba0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f68b4006fd0 tx=0x7f68b4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:00.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.755+0000 7f68c0ff9700 1 -- 172.21.15.46:0/2883308492 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f68b8022d50 con 0x7f68c41065e0 2024-04-27T03:55:00.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.973+0000 7f68c9b49700 1 -- 172.21.15.46:0/2883308492 --> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f68c4102df0 con 0x7f68ac0386e0 2024-04-27T03:55:00.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.979+0000 7f68c0ff9700 1 -- 172.21.15.46:0/2883308492 <== mgr.14118 v2:172.21.15.46:6800/333567692 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f68c4102df0 con 0x7f68ac0386e0 2024-04-27T03:55:00.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.981+0000 7f68c9b49700 1 -- 172.21.15.46:0/2883308492 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f68ac0386e0 msgr2=0x7f68ac03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:00.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.981+0000 7f68c9b49700 1 --2- 172.21.15.46:0/2883308492 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f68ac0386e0 0x7f68ac03aba0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f68b4006fd0 tx=0x7f68b4006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:00.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.981+0000 7f68c9b49700 1 -- 172.21.15.46:0/2883308492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c41065e0 msgr2=0x7f68c419a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:00.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.981+0000 7f68c9b49700 1 --2- 172.21.15.46:0/2883308492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c41065e0 0x7f68c419a340 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f68b8009fb0 tx=0x7f68b8011890 comp rx=0 tx=0).stop 2024-04-27T03:55:00.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.981+0000 7f68c9b49700 1 -- 172.21.15.46:0/2883308492 shutdown_connections 2024-04-27T03:55:00.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.981+0000 7f68c9b49700 1 --2- 172.21.15.46:0/2883308492 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f68ac0386e0 0x7f68ac03aba0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:00.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.981+0000 7f68c9b49700 1 --2- 172.21.15.46:0/2883308492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c41065e0 0x7f68c419a340 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:00.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.981+0000 7f68c9b49700 1 -- 172.21.15.46:0/2883308492 >> 172.21.15.46:0/2883308492 conn(0x7f68c4101b80 msgr2=0x7f68c41026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:00.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.981+0000 7f68c9b49700 1 -- 172.21.15.46:0/2883308492 shutdown_connections 2024-04-27T03:55:00.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:54:59.981+0000 7f68c9b49700 1 -- 172.21.15.46:0/2883308492 wait complete. 2024-04-27T03:55:00.567 INFO:teuthology.orchestra.run.smithi046.stdout:Deploying node-exporter service with default placement... 2024-04-27T03:55:01.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:00 smithi046 ceph-mon[26899]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:55:01.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:00 smithi046 ceph-mon[26899]: Saving service grafana spec with placement count:1 2024-04-27T03:55:01.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:00 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:01.836 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-27T03:55:01.836 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.050+0000 7f0b5fffd700 1 Processor -- start 2024-04-27T03:55:01.837 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.050+0000 7f0b5fffd700 1 -- start start 2024-04-27T03:55:01.837 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.050+0000 7f0b5fffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b581065f0 0x7f0b58106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:01.837 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.050+0000 7f0b5fffd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b58106fe0 con 0x7f0b581065f0 2024-04-27T03:55:01.837 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.051+0000 7f0b5dd99700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b581065f0 0x7f0b58106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:01.837 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.051+0000 7f0b5dd99700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b581065f0 0x7f0b58106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55238/0 (socket says 172.21.15.46:55238) 2024-04-27T03:55:01.837 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.051+0000 7f0b5dd99700 1 -- 172.21.15.46:0/3289995888 learned_addr learned my addr 172.21.15.46:0/3289995888 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:01.837 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.051+0000 7f0b5dd99700 1 -- 172.21.15.46:0/3289995888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b581077a0 con 0x7f0b581065f0 2024-04-27T03:55:01.837 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.051+0000 7f0b5dd99700 1 --2- 172.21.15.46:0/3289995888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b581065f0 0x7f0b58106a10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009e80 tx=0x7f0b5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40daa3e3a1121362 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:01.837 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.051+0000 7f0b5cd97700 1 -- 172.21.15.46:0/3289995888 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b5000e070 con 0x7f0b581065f0 2024-04-27T03:55:01.837 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.051+0000 7f0b5cd97700 1 -- 172.21.15.46:0/3289995888 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b5000b8c0 con 0x7f0b581065f0 2024-04-27T03:55:01.837 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.051+0000 7f0b5cd97700 1 -- 172.21.15.46:0/3289995888 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b500044d0 con 0x7f0b581065f0 2024-04-27T03:55:01.837 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.052+0000 7f0b5fffd700 1 -- 172.21.15.46:0/3289995888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b581065f0 msgr2=0x7f0b58106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:01.837 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.052+0000 7f0b5fffd700 1 --2- 172.21.15.46:0/3289995888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b581065f0 0x7f0b58106a10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009e80 tx=0x7f0b5000b330 comp rx=0 tx=0).stop 2024-04-27T03:55:01.837 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.052+0000 7f0b5fffd700 1 -- 172.21.15.46:0/3289995888 shutdown_connections 2024-04-27T03:55:01.838 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.052+0000 7f0b5fffd700 1 --2- 172.21.15.46:0/3289995888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b581065f0 0x7f0b58106a10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:01.838 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.052+0000 7f0b5fffd700 1 -- 172.21.15.46:0/3289995888 >> 172.21.15.46:0/3289995888 conn(0x7f0b58101b50 msgr2=0x7f0b58103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:01.838 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.052+0000 7f0b5fffd700 1 -- 172.21.15.46:0/3289995888 shutdown_connections 2024-04-27T03:55:01.838 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.052+0000 7f0b5fffd700 1 -- 172.21.15.46:0/3289995888 wait complete. 2024-04-27T03:55:01.838 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.053+0000 7f0b5fffd700 1 Processor -- start 2024-04-27T03:55:01.838 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.053+0000 7f0b5fffd700 1 -- start start 2024-04-27T03:55:01.838 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.053+0000 7f0b5fffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b581065f0 0x7f0b5819a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:01.838 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.053+0000 7f0b5fffd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5819a890 con 0x7f0b581065f0 2024-04-27T03:55:01.838 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.053+0000 7f0b5dd99700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b581065f0 0x7f0b5819a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:01.838 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.053+0000 7f0b5dd99700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b581065f0 0x7f0b5819a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55250/0 (socket says 172.21.15.46:55250) 2024-04-27T03:55:01.838 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.053+0000 7f0b5dd99700 1 -- 172.21.15.46:0/1992289039 learned_addr learned my addr 172.21.15.46:0/1992289039 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:01.838 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.053+0000 7f0b5dd99700 1 -- 172.21.15.46:0/1992289039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b50009b30 con 0x7f0b581065f0 2024-04-27T03:55:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.053+0000 7f0b5dd99700 1 --2- 172.21.15.46:0/1992289039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b581065f0 0x7f0b5819a350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0b50000f80 tx=0x7f0b5000bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.053+0000 7f0b56ffd700 1 -- 172.21.15.46:0/1992289039 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b5000e040 con 0x7f0b581065f0 2024-04-27T03:55:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.054+0000 7f0b5fffd700 1 -- 172.21.15.46:0/1992289039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5819aa90 con 0x7f0b581065f0 2024-04-27T03:55:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.054+0000 7f0b56ffd700 1 -- 172.21.15.46:0/1992289039 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b50003c30 con 0x7f0b581065f0 2024-04-27T03:55:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.054+0000 7f0b56ffd700 1 -- 172.21.15.46:0/1992289039 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b500125e0 con 0x7f0b581065f0 2024-04-27T03:55:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.054+0000 7f0b5fffd700 1 -- 172.21.15.46:0/1992289039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5819af90 con 0x7f0b581065f0 2024-04-27T03:55:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.054+0000 7f0b56ffd700 1 -- 172.21.15.46:0/1992289039 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f0b50019040 con 0x7f0b581065f0 2024-04-27T03:55:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.054+0000 7f0b5fffd700 1 -- 172.21.15.46:0/1992289039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b58062510 con 0x7f0b581065f0 2024-04-27T03:55:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.055+0000 7f0b56ffd700 1 --2- 172.21.15.46:0/1992289039 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f0b40038720 0x7f0b4003abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.055+0000 7f0b56ffd700 1 -- 172.21.15.46:0/1992289039 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0b5004c2c0 con 0x7f0b581065f0 2024-04-27T03:55:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.055+0000 7f0b5d598700 1 --2- 172.21.15.46:0/1992289039 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f0b40038720 0x7f0b4003abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.055+0000 7f0b5d598700 1 --2- 172.21.15.46:0/1992289039 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f0b40038720 0x7f0b4003abe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c006fd0 tx=0x7f0b4c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.059+0000 7f0b56ffd700 1 -- 172.21.15.46:0/1992289039 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f0b5002cb40 con 0x7f0b581065f0 2024-04-27T03:55:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.274+0000 7f0b5fffd700 1 -- 172.21.15.46:0/1992289039 --> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b5810a980 con 0x7f0b40038720 2024-04-27T03:55:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.279+0000 7f0b56ffd700 1 -- 172.21.15.46:0/1992289039 <== mgr.14118 v2:172.21.15.46:6800/333567692 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f0b5810a980 con 0x7f0b40038720 2024-04-27T03:55:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.281+0000 7f0b5fffd700 1 -- 172.21.15.46:0/1992289039 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f0b40038720 msgr2=0x7f0b4003abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.281+0000 7f0b5fffd700 1 --2- 172.21.15.46:0/1992289039 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f0b40038720 0x7f0b4003abe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c006fd0 tx=0x7f0b4c006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.281+0000 7f0b5fffd700 1 -- 172.21.15.46:0/1992289039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b581065f0 msgr2=0x7f0b5819a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.281+0000 7f0b5fffd700 1 --2- 172.21.15.46:0/1992289039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b581065f0 0x7f0b5819a350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0b50000f80 tx=0x7f0b5000bcf0 comp rx=0 tx=0).stop 2024-04-27T03:55:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.281+0000 7f0b5fffd700 1 -- 172.21.15.46:0/1992289039 shutdown_connections 2024-04-27T03:55:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.281+0000 7f0b5fffd700 1 --2- 172.21.15.46:0/1992289039 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f0b40038720 0x7f0b4003abe0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.281+0000 7f0b5fffd700 1 --2- 172.21.15.46:0/1992289039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b581065f0 0x7f0b5819a350 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.281+0000 7f0b5fffd700 1 -- 172.21.15.46:0/1992289039 >> 172.21.15.46:0/1992289039 conn(0x7f0b58101b50 msgr2=0x7f0b58102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:01.841 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.281+0000 7f0b5fffd700 1 -- 172.21.15.46:0/1992289039 shutdown_connections 2024-04-27T03:55:01.841 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:01.281+0000 7f0b5fffd700 1 -- 172.21.15.46:0/1992289039 wait complete. 2024-04-27T03:55:01.841 INFO:teuthology.orchestra.run.smithi046.stdout:Deploying alertmanager service with default placement... 2024-04-27T03:55:02.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:02 smithi046 ceph-mon[26899]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:55:02.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:02 smithi046 ceph-mon[26899]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:55:02.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:02 smithi046 ceph-mon[26899]: Saving service node-exporter spec with placement * 2024-04-27T03:55:02.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:02 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:03.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-27T03:55:03.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.282+0000 7ff6387f9700 1 Processor -- start 2024-04-27T03:55:03.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.282+0000 7ff6387f9700 1 -- start start 2024-04-27T03:55:03.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.282+0000 7ff6387f9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff630107020 0x7ff630109410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:03.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.282+0000 7ff6387f9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff630079ed0 con 0x7ff630107020 2024-04-27T03:55:03.111 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.282+0000 7ff636595700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff630107020 0x7ff630109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:03.111 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.282+0000 7ff636595700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff630107020 0x7ff630109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55258/0 (socket says 172.21.15.46:55258) 2024-04-27T03:55:03.111 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.282+0000 7ff636595700 1 -- 172.21.15.46:0/3366695655 learned_addr learned my addr 172.21.15.46:0/3366695655 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:03.111 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.283+0000 7ff636595700 1 -- 172.21.15.46:0/3366695655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff630109950 con 0x7ff630107020 2024-04-27T03:55:03.111 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.283+0000 7ff636595700 1 --2- 172.21.15.46:0/3366695655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff630107020 0x7ff630109410 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff6200098a0 tx=0x7ff620009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b3c400b144329c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:03.111 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.283+0000 7ff635593700 1 -- 172.21.15.46:0/3366695655 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff620004180 con 0x7ff630107020 2024-04-27T03:55:03.111 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.283+0000 7ff635593700 1 -- 172.21.15.46:0/3366695655 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6200042e0 con 0x7ff630107020 2024-04-27T03:55:03.111 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.283+0000 7ff635593700 1 -- 172.21.15.46:0/3366695655 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff620011b10 con 0x7ff630107020 2024-04-27T03:55:03.111 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.284+0000 7ff6387f9700 1 -- 172.21.15.46:0/3366695655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff630107020 msgr2=0x7ff630109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:03.111 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.284+0000 7ff6387f9700 1 --2- 172.21.15.46:0/3366695655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff630107020 0x7ff630109410 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff6200098a0 tx=0x7ff620009bb0 comp rx=0 tx=0).stop 2024-04-27T03:55:03.111 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.284+0000 7ff6387f9700 1 -- 172.21.15.46:0/3366695655 shutdown_connections 2024-04-27T03:55:03.111 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.284+0000 7ff6387f9700 1 --2- 172.21.15.46:0/3366695655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff630107020 0x7ff630109410 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:03.112 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.284+0000 7ff6387f9700 1 -- 172.21.15.46:0/3366695655 >> 172.21.15.46:0/3366695655 conn(0x7ff630100ab0 msgr2=0x7ff630102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:03.112 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.284+0000 7ff6387f9700 1 -- 172.21.15.46:0/3366695655 shutdown_connections 2024-04-27T03:55:03.112 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.284+0000 7ff6387f9700 1 -- 172.21.15.46:0/3366695655 wait complete. 2024-04-27T03:55:03.112 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.284+0000 7ff6387f9700 1 Processor -- start 2024-04-27T03:55:03.112 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.285+0000 7ff6387f9700 1 -- start start 2024-04-27T03:55:03.112 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.285+0000 7ff6387f9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff630107020 0x7ff63019c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:03.112 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.285+0000 7ff6387f9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63019ca60 con 0x7ff630107020 2024-04-27T03:55:03.112 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.285+0000 7ff636595700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff630107020 0x7ff63019c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:03.112 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.285+0000 7ff636595700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff630107020 0x7ff63019c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55270/0 (socket says 172.21.15.46:55270) 2024-04-27T03:55:03.112 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.285+0000 7ff636595700 1 -- 172.21.15.46:0/3487719743 learned_addr learned my addr 172.21.15.46:0/3487719743 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:03.112 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.285+0000 7ff636595700 1 -- 172.21.15.46:0/3487719743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff620009550 con 0x7ff630107020 2024-04-27T03:55:03.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.285+0000 7ff636595700 1 --2- 172.21.15.46:0/3487719743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff630107020 0x7ff63019c520 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff6200117b0 tx=0x7ff620011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:03.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.285+0000 7ff6277fe700 1 -- 172.21.15.46:0/3487719743 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6200038b0 con 0x7ff630107020 2024-04-27T03:55:03.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.285+0000 7ff6277fe700 1 -- 172.21.15.46:0/3487719743 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff620003a10 con 0x7ff630107020 2024-04-27T03:55:03.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.285+0000 7ff6387f9700 1 -- 172.21.15.46:0/3487719743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63019cc60 con 0x7ff630107020 2024-04-27T03:55:03.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.286+0000 7ff6277fe700 1 -- 172.21.15.46:0/3487719743 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff620024780 con 0x7ff630107020 2024-04-27T03:55:03.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.286+0000 7ff6387f9700 1 -- 172.21.15.46:0/3487719743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63019d100 con 0x7ff630107020 2024-04-27T03:55:03.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.286+0000 7ff6277fe700 1 -- 172.21.15.46:0/3487719743 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7ff6200249c0 con 0x7ff630107020 2024-04-27T03:55:03.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.286+0000 7ff6277fe700 1 --2- 172.21.15.46:0/3487719743 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7ff61c038770 0x7ff61c03ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:03.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.286+0000 7ff6387f9700 1 -- 172.21.15.46:0/3487719743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63010a740 con 0x7ff630107020 2024-04-27T03:55:03.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.287+0000 7ff6277fe700 1 -- 172.21.15.46:0/3487719743 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff62004c9d0 con 0x7ff630107020 2024-04-27T03:55:03.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.287+0000 7ff635d94700 1 --2- 172.21.15.46:0/3487719743 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7ff61c038770 0x7ff61c03ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:03.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.287+0000 7ff635d94700 1 --2- 172.21.15.46:0/3487719743 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7ff61c038770 0x7ff61c03ac30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff62c006fd0 tx=0x7ff62c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:03.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.291+0000 7ff6277fe700 1 -- 172.21.15.46:0/3487719743 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7ff620024c70 con 0x7ff630107020 2024-04-27T03:55:03.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.509+0000 7ff6387f9700 1 -- 172.21.15.46:0/3487719743 --> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7ff630101c40 con 0x7ff61c038770 2024-04-27T03:55:03.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.514+0000 7ff6277fe700 1 -- 172.21.15.46:0/3487719743 <== mgr.14118 v2:172.21.15.46:6800/333567692 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7ff630101c40 con 0x7ff61c038770 2024-04-27T03:55:03.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.516+0000 7ff6387f9700 1 -- 172.21.15.46:0/3487719743 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7ff61c038770 msgr2=0x7ff61c03ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:03.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.516+0000 7ff6387f9700 1 --2- 172.21.15.46:0/3487719743 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7ff61c038770 0x7ff61c03ac30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff62c006fd0 tx=0x7ff62c006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:03.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.516+0000 7ff6387f9700 1 -- 172.21.15.46:0/3487719743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff630107020 msgr2=0x7ff63019c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:03.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.516+0000 7ff6387f9700 1 --2- 172.21.15.46:0/3487719743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff630107020 0x7ff63019c520 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff6200117b0 tx=0x7ff620011890 comp rx=0 tx=0).stop 2024-04-27T03:55:03.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.516+0000 7ff6387f9700 1 -- 172.21.15.46:0/3487719743 shutdown_connections 2024-04-27T03:55:03.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.516+0000 7ff6387f9700 1 --2- 172.21.15.46:0/3487719743 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7ff61c038770 0x7ff61c03ac30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:03.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.516+0000 7ff6387f9700 1 --2- 172.21.15.46:0/3487719743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff630107020 0x7ff63019c520 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:03.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.516+0000 7ff6387f9700 1 -- 172.21.15.46:0/3487719743 >> 172.21.15.46:0/3487719743 conn(0x7ff630100ab0 msgr2=0x7ff630101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:03.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.516+0000 7ff6387f9700 1 -- 172.21.15.46:0/3487719743 shutdown_connections 2024-04-27T03:55:03.115 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:02.516+0000 7ff6387f9700 1 -- 172.21.15.46:0/3487719743 wait complete. 2024-04-27T03:55:03.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:03 smithi046 ceph-mon[26899]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:55:03.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:03 smithi046 ceph-mon[26899]: Saving service alertmanager spec with placement count:1 2024-04-27T03:55:03.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:03 smithi046 ceph-mon[26899]: from='mgr.14118 172.21.15.46:0/1303213767' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:04.250 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.527+0000 7f21a6e54700 1 Processor -- start 2024-04-27T03:55:04.250 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.527+0000 7f21a6e54700 1 -- start start 2024-04-27T03:55:04.250 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.527+0000 7f21a6e54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a007bec0 0x7f21a007a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:04.250 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.527+0000 7f21a6e54700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a007a8f0 con 0x7f21a007bec0 2024-04-27T03:55:04.250 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.528+0000 7f21a4bf0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a007bec0 0x7f21a007a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:04.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.528+0000 7f21a4bf0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a007bec0 0x7f21a007a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55280/0 (socket says 172.21.15.46:55280) 2024-04-27T03:55:04.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.528+0000 7f21a4bf0700 1 -- 172.21.15.46:0/3427872433 learned_addr learned my addr 172.21.15.46:0/3427872433 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:04.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.528+0000 7f21a4bf0700 1 -- 172.21.15.46:0/3427872433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a007aa30 con 0x7f21a007bec0 2024-04-27T03:55:04.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.528+0000 7f21a4bf0700 1 --2- 172.21.15.46:0/3427872433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a007bec0 0x7f21a007a320 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f21940098a0 tx=0x7f2194009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf9271d5f0cda2e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:04.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.528+0000 7f219f7fe700 1 -- 172.21.15.46:0/3427872433 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2194004180 con 0x7f21a007bec0 2024-04-27T03:55:04.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.528+0000 7f219f7fe700 1 -- 172.21.15.46:0/3427872433 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21940042e0 con 0x7f21a007bec0 2024-04-27T03:55:04.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.528+0000 7f219f7fe700 1 -- 172.21.15.46:0/3427872433 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2194011b10 con 0x7f21a007bec0 2024-04-27T03:55:04.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.529+0000 7f21a6e54700 1 -- 172.21.15.46:0/3427872433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a007bec0 msgr2=0x7f21a007a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:04.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.529+0000 7f21a6e54700 1 --2- 172.21.15.46:0/3427872433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a007bec0 0x7f21a007a320 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f21940098a0 tx=0x7f2194009bb0 comp rx=0 tx=0).stop 2024-04-27T03:55:04.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.529+0000 7f21a6e54700 1 -- 172.21.15.46:0/3427872433 shutdown_connections 2024-04-27T03:55:04.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.529+0000 7f21a6e54700 1 --2- 172.21.15.46:0/3427872433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a007bec0 0x7f21a007a320 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:04.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.529+0000 7f21a6e54700 1 -- 172.21.15.46:0/3427872433 >> 172.21.15.46:0/3427872433 conn(0x7f21a0101be0 msgr2=0x7f21a0104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:04.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.529+0000 7f21a6e54700 1 -- 172.21.15.46:0/3427872433 shutdown_connections 2024-04-27T03:55:04.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.529+0000 7f21a6e54700 1 -- 172.21.15.46:0/3427872433 wait complete. 2024-04-27T03:55:04.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.530+0000 7f21a6e54700 1 Processor -- start 2024-04-27T03:55:04.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.530+0000 7f21a6e54700 1 -- start start 2024-04-27T03:55:04.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.530+0000 7f21a6e54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a007bec0 0x7f21a019c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:04.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.530+0000 7f21a6e54700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a019cab0 con 0x7f21a007bec0 2024-04-27T03:55:04.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.530+0000 7f21a4bf0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a007bec0 0x7f21a019c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:04.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.530+0000 7f21a4bf0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a007bec0 0x7f21a019c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55282/0 (socket says 172.21.15.46:55282) 2024-04-27T03:55:04.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.530+0000 7f21a4bf0700 1 -- 172.21.15.46:0/1291433261 learned_addr learned my addr 172.21.15.46:0/1291433261 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:04.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.530+0000 7f21a4bf0700 1 -- 172.21.15.46:0/1291433261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2194009550 con 0x7f21a007bec0 2024-04-27T03:55:04.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.531+0000 7f21a4bf0700 1 --2- 172.21.15.46:0/1291433261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a007bec0 0x7f21a019c570 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f21940117b0 tx=0x7f2194011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:04.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.531+0000 7f219dffb700 1 -- 172.21.15.46:0/1291433261 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21940038b0 con 0x7f21a007bec0 2024-04-27T03:55:04.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.531+0000 7f21a6e54700 1 -- 172.21.15.46:0/1291433261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a019ccb0 con 0x7f21a007bec0 2024-04-27T03:55:04.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.531+0000 7f219dffb700 1 -- 172.21.15.46:0/1291433261 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2194003a10 con 0x7f21a007bec0 2024-04-27T03:55:04.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.531+0000 7f219dffb700 1 -- 172.21.15.46:0/1291433261 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21940246e0 con 0x7f21a007bec0 2024-04-27T03:55:04.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.531+0000 7f21a6e54700 1 -- 172.21.15.46:0/1291433261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a019d150 con 0x7f21a007bec0 2024-04-27T03:55:04.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.532+0000 7f219dffb700 1 -- 172.21.15.46:0/1291433261 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f219402d3f0 con 0x7f21a007bec0 2024-04-27T03:55:04.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.532+0000 7f21a6e54700 1 -- 172.21.15.46:0/1291433261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a010c980 con 0x7f21a007bec0 2024-04-27T03:55:04.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.532+0000 7f219dffb700 1 --2- 172.21.15.46:0/1291433261 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f2190038720 0x7f219003abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:04.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.532+0000 7f219dffb700 1 -- 172.21.15.46:0/1291433261 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f219404c9f0 con 0x7f21a007bec0 2024-04-27T03:55:04.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.532+0000 7f219ffff700 1 --2- 172.21.15.46:0/1291433261 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f2190038720 0x7f219003abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:04.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.533+0000 7f219ffff700 1 --2- 172.21.15.46:0/1291433261 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f2190038720 0x7f219003abe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f218c006fd0 tx=0x7f218c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:04.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.537+0000 7f219dffb700 1 -- 172.21.15.46:0/1291433261 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f2194018c90 con 0x7f21a007bec0 2024-04-27T03:55:04.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.739+0000 7f21a6e54700 1 -- 172.21.15.46:0/1291433261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f21a010cbc0 con 0x7f21a007bec0 2024-04-27T03:55:04.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.742+0000 7f219dffb700 1 -- 172.21.15.46:0/1291433261 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f2194018ab0 con 0x7f21a007bec0 2024-04-27T03:55:04.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.745+0000 7f21a6e54700 1 -- 172.21.15.46:0/1291433261 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f2190038720 msgr2=0x7f219003abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:04.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.745+0000 7f21a6e54700 1 --2- 172.21.15.46:0/1291433261 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f2190038720 0x7f219003abe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f218c006fd0 tx=0x7f218c006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:04.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.745+0000 7f21a6e54700 1 -- 172.21.15.46:0/1291433261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a007bec0 msgr2=0x7f21a019c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:04.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.745+0000 7f21a6e54700 1 --2- 172.21.15.46:0/1291433261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a007bec0 0x7f21a019c570 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f21940117b0 tx=0x7f2194011890 comp rx=0 tx=0).stop 2024-04-27T03:55:04.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.745+0000 7f21a6e54700 1 -- 172.21.15.46:0/1291433261 shutdown_connections 2024-04-27T03:55:04.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.745+0000 7f21a6e54700 1 --2- 172.21.15.46:0/1291433261 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f2190038720 0x7f219003abe0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:04.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.745+0000 7f21a6e54700 1 --2- 172.21.15.46:0/1291433261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a007bec0 0x7f21a019c570 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:04.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.745+0000 7f21a6e54700 1 -- 172.21.15.46:0/1291433261 >> 172.21.15.46:0/1291433261 conn(0x7f21a0101be0 msgr2=0x7f21a01026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:04.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.745+0000 7f21a6e54700 1 -- 172.21.15.46:0/1291433261 shutdown_connections 2024-04-27T03:55:04.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:03.745+0000 7f21a6e54700 1 -- 172.21.15.46:0/1291433261 wait complete. 2024-04-27T03:55:04.894 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:04 smithi046 ceph-mon[26899]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:55:04.894 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:04 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/1291433261' entity='client.admin' 2024-04-27T03:55:05.613 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.682+0000 7fd0678b8700 1 Processor -- start 2024-04-27T03:55:05.613 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.682+0000 7fd0678b8700 1 -- start start 2024-04-27T03:55:05.613 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.682+0000 7fd0678b8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0601065f0 0x7fd060106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:05.613 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.682+0000 7fd0678b8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd060106fe0 con 0x7fd0601065f0 2024-04-27T03:55:05.613 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.683+0000 7fd065654700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0601065f0 0x7fd060106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:05.613 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.683+0000 7fd065654700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0601065f0 0x7fd060106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55286/0 (socket says 172.21.15.46:55286) 2024-04-27T03:55:05.614 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.683+0000 7fd065654700 1 -- 172.21.15.46:0/3403766704 learned_addr learned my addr 172.21.15.46:0/3403766704 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:05.614 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.683+0000 7fd065654700 1 -- 172.21.15.46:0/3403766704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0601077a0 con 0x7fd0601065f0 2024-04-27T03:55:05.614 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.683+0000 7fd065654700 1 --2- 172.21.15.46:0/3403766704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0601065f0 0x7fd060106a10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd058009e80 tx=0x7fd05800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b722c4bf3abda6e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:05.614 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.683+0000 7fd05ffff700 1 -- 172.21.15.46:0/3403766704 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd05800e070 con 0x7fd0601065f0 2024-04-27T03:55:05.614 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.683+0000 7fd05ffff700 1 -- 172.21.15.46:0/3403766704 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd05800b8c0 con 0x7fd0601065f0 2024-04-27T03:55:05.614 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.683+0000 7fd05ffff700 1 -- 172.21.15.46:0/3403766704 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0580044d0 con 0x7fd0601065f0 2024-04-27T03:55:05.614 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.684+0000 7fd0678b8700 1 -- 172.21.15.46:0/3403766704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0601065f0 msgr2=0x7fd060106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:05.614 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.684+0000 7fd0678b8700 1 --2- 172.21.15.46:0/3403766704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0601065f0 0x7fd060106a10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd058009e80 tx=0x7fd05800b330 comp rx=0 tx=0).stop 2024-04-27T03:55:05.614 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.684+0000 7fd0678b8700 1 -- 172.21.15.46:0/3403766704 shutdown_connections 2024-04-27T03:55:05.614 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.684+0000 7fd0678b8700 1 --2- 172.21.15.46:0/3403766704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0601065f0 0x7fd060106a10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:05.614 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.684+0000 7fd0678b8700 1 -- 172.21.15.46:0/3403766704 >> 172.21.15.46:0/3403766704 conn(0x7fd060101bd0 msgr2=0x7fd060103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:05.614 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.684+0000 7fd0678b8700 1 -- 172.21.15.46:0/3403766704 shutdown_connections 2024-04-27T03:55:05.614 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.684+0000 7fd0678b8700 1 -- 172.21.15.46:0/3403766704 wait complete. 2024-04-27T03:55:05.615 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.684+0000 7fd0678b8700 1 Processor -- start 2024-04-27T03:55:05.615 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.685+0000 7fd0678b8700 1 -- start start 2024-04-27T03:55:05.615 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.685+0000 7fd0678b8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0601065f0 0x7fd06019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:05.615 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.685+0000 7fd0678b8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06019a7f0 con 0x7fd0601065f0 2024-04-27T03:55:05.615 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.685+0000 7fd065654700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0601065f0 0x7fd06019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:05.615 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.685+0000 7fd065654700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0601065f0 0x7fd06019a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55292/0 (socket says 172.21.15.46:55292) 2024-04-27T03:55:05.615 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.685+0000 7fd065654700 1 -- 172.21.15.46:0/1687557790 learned_addr learned my addr 172.21.15.46:0/1687557790 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:05.615 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.685+0000 7fd065654700 1 -- 172.21.15.46:0/1687557790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd058009b30 con 0x7fd0601065f0 2024-04-27T03:55:05.615 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.685+0000 7fd065654700 1 --2- 172.21.15.46:0/1687557790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0601065f0 0x7fd06019a2b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd0580072a0 tx=0x7fd05800bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:05.615 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.685+0000 7fd05e7fc700 1 -- 172.21.15.46:0/1687557790 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd05800e040 con 0x7fd0601065f0 2024-04-27T03:55:05.615 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.685+0000 7fd05e7fc700 1 -- 172.21.15.46:0/1687557790 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd058003dc0 con 0x7fd0601065f0 2024-04-27T03:55:05.615 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.685+0000 7fd0678b8700 1 -- 172.21.15.46:0/1687557790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd06019a9f0 con 0x7fd0601065f0 2024-04-27T03:55:05.615 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.685+0000 7fd05e7fc700 1 -- 172.21.15.46:0/1687557790 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd058012600 con 0x7fd0601065f0 2024-04-27T03:55:05.616 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.685+0000 7fd0678b8700 1 -- 172.21.15.46:0/1687557790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd06019aef0 con 0x7fd0601065f0 2024-04-27T03:55:05.616 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.686+0000 7fd05e7fc700 1 -- 172.21.15.46:0/1687557790 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fd058019040 con 0x7fd0601065f0 2024-04-27T03:55:05.616 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.686+0000 7fd0678b8700 1 -- 172.21.15.46:0/1687557790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd06010a9d0 con 0x7fd0601065f0 2024-04-27T03:55:05.616 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.686+0000 7fd05e7fc700 1 --2- 172.21.15.46:0/1687557790 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fd0480386c0 0x7fd04803ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:05.616 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.687+0000 7fd05e7fc700 1 -- 172.21.15.46:0/1687557790 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd05804c2d0 con 0x7fd0601065f0 2024-04-27T03:55:05.616 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.687+0000 7fd064e53700 1 --2- 172.21.15.46:0/1687557790 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fd0480386c0 0x7fd04803ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:05.616 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.687+0000 7fd064e53700 1 --2- 172.21.15.46:0/1687557790 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fd0480386c0 0x7fd04803ab80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd054006fd0 tx=0x7fd054006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:05.616 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.691+0000 7fd05e7fc700 1 -- 172.21.15.46:0/1687557790 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fd05801a9e0 con 0x7fd0601065f0 2024-04-27T03:55:05.616 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.891+0000 7fd0678b8700 1 -- 172.21.15.46:0/1687557790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fd06019d720 con 0x7fd0601065f0 2024-04-27T03:55:05.616 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.894+0000 7fd05e7fc700 1 -- 172.21.15.46:0/1687557790 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fd05801f020 con 0x7fd0601065f0 2024-04-27T03:55:05.616 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.896+0000 7fd0678b8700 1 -- 172.21.15.46:0/1687557790 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fd0480386c0 msgr2=0x7fd04803ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:05.616 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.896+0000 7fd0678b8700 1 --2- 172.21.15.46:0/1687557790 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fd0480386c0 0x7fd04803ab80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd054006fd0 tx=0x7fd054006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:05.616 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.896+0000 7fd0678b8700 1 -- 172.21.15.46:0/1687557790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0601065f0 msgr2=0x7fd06019a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:05.617 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.896+0000 7fd0678b8700 1 --2- 172.21.15.46:0/1687557790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0601065f0 0x7fd06019a2b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd0580072a0 tx=0x7fd05800bf70 comp rx=0 tx=0).stop 2024-04-27T03:55:05.617 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.896+0000 7fd0678b8700 1 -- 172.21.15.46:0/1687557790 shutdown_connections 2024-04-27T03:55:05.617 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.896+0000 7fd0678b8700 1 --2- 172.21.15.46:0/1687557790 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fd0480386c0 0x7fd04803ab80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:05.617 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.896+0000 7fd0678b8700 1 --2- 172.21.15.46:0/1687557790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0601065f0 0x7fd06019a2b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:05.617 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.896+0000 7fd0678b8700 1 -- 172.21.15.46:0/1687557790 >> 172.21.15.46:0/1687557790 conn(0x7fd060101bd0 msgr2=0x7fd0601026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:05.617 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.896+0000 7fd0678b8700 1 -- 172.21.15.46:0/1687557790 shutdown_connections 2024-04-27T03:55:05.617 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:04.896+0000 7fd0678b8700 1 -- 172.21.15.46:0/1687557790 wait complete. 2024-04-27T03:55:05.617 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling the dashboard module... 2024-04-27T03:55:06.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:05 smithi046 ceph-mon[26899]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:55:06.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:05 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/1687557790' entity='client.admin' 2024-04-27T03:55:07.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:06 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/512705936' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-27T03:55:07.515 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.048+0000 7f9d2cdef700 1 Processor -- start 2024-04-27T03:55:07.515 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.048+0000 7f9d2cdef700 1 -- start start 2024-04-27T03:55:07.515 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.049+0000 7f9d2cdef700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d281065f0 0x7f9d28106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:07.515 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.049+0000 7f9d2cdef700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d28106fe0 con 0x7f9d281065f0 2024-04-27T03:55:07.516 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.049+0000 7f9d2659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d281065f0 0x7f9d28106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:07.516 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.049+0000 7f9d2659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d281065f0 0x7f9d28106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55308/0 (socket says 172.21.15.46:55308) 2024-04-27T03:55:07.516 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.049+0000 7f9d2659c700 1 -- 172.21.15.46:0/3518134877 learned_addr learned my addr 172.21.15.46:0/3518134877 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:07.516 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.049+0000 7f9d2659c700 1 -- 172.21.15.46:0/3518134877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d281077a0 con 0x7f9d281065f0 2024-04-27T03:55:07.516 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.049+0000 7f9d2659c700 1 --2- 172.21.15.46:0/3518134877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d281065f0 0x7f9d28106a10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9d100098a0 tx=0x7f9d10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db8fee479e8d2a85 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:07.516 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.049+0000 7f9d2559a700 1 -- 172.21.15.46:0/3518134877 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d10004180 con 0x7f9d281065f0 2024-04-27T03:55:07.516 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.049+0000 7f9d2559a700 1 -- 172.21.15.46:0/3518134877 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d100042e0 con 0x7f9d281065f0 2024-04-27T03:55:07.516 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.049+0000 7f9d2559a700 1 -- 172.21.15.46:0/3518134877 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d10011b10 con 0x7f9d281065f0 2024-04-27T03:55:07.516 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.050+0000 7f9d2cdef700 1 -- 172.21.15.46:0/3518134877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d281065f0 msgr2=0x7f9d28106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:07.516 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.050+0000 7f9d2cdef700 1 --2- 172.21.15.46:0/3518134877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d281065f0 0x7f9d28106a10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9d100098a0 tx=0x7f9d10009bb0 comp rx=0 tx=0).stop 2024-04-27T03:55:07.516 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.050+0000 7f9d2cdef700 1 -- 172.21.15.46:0/3518134877 shutdown_connections 2024-04-27T03:55:07.516 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.050+0000 7f9d2cdef700 1 --2- 172.21.15.46:0/3518134877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d281065f0 0x7f9d28106a10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:07.516 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.050+0000 7f9d2cdef700 1 -- 172.21.15.46:0/3518134877 >> 172.21.15.46:0/3518134877 conn(0x7f9d28101bd0 msgr2=0x7f9d28103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:07.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.050+0000 7f9d2cdef700 1 -- 172.21.15.46:0/3518134877 shutdown_connections 2024-04-27T03:55:07.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.050+0000 7f9d2cdef700 1 -- 172.21.15.46:0/3518134877 wait complete. 2024-04-27T03:55:07.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.051+0000 7f9d2cdef700 1 Processor -- start 2024-04-27T03:55:07.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.051+0000 7f9d2cdef700 1 -- start start 2024-04-27T03:55:07.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.051+0000 7f9d2cdef700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d281065f0 0x7f9d2807baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:07.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.051+0000 7f9d2cdef700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2807c030 con 0x7f9d281065f0 2024-04-27T03:55:07.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.051+0000 7f9d2659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d281065f0 0x7f9d2807baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:07.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.051+0000 7f9d2659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d281065f0 0x7f9d2807baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55324/0 (socket says 172.21.15.46:55324) 2024-04-27T03:55:07.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.051+0000 7f9d2659c700 1 -- 172.21.15.46:0/512705936 learned_addr learned my addr 172.21.15.46:0/512705936 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:07.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.052+0000 7f9d2659c700 1 -- 172.21.15.46:0/512705936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d10009550 con 0x7f9d281065f0 2024-04-27T03:55:07.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.052+0000 7f9d2659c700 1 --2- 172.21.15.46:0/512705936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d281065f0 0x7f9d2807baf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9d100117b0 tx=0x7f9d10011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:07.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.052+0000 7f9d1f7fe700 1 -- 172.21.15.46:0/512705936 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d100038b0 con 0x7f9d281065f0 2024-04-27T03:55:07.517 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.052+0000 7f9d2cdef700 1 -- 172.21.15.46:0/512705936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d2807a2c0 con 0x7f9d281065f0 2024-04-27T03:55:07.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.052+0000 7f9d1f7fe700 1 -- 172.21.15.46:0/512705936 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d10003a10 con 0x7f9d281065f0 2024-04-27T03:55:07.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.052+0000 7f9d1f7fe700 1 -- 172.21.15.46:0/512705936 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d100246e0 con 0x7f9d281065f0 2024-04-27T03:55:07.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.052+0000 7f9d2cdef700 1 -- 172.21.15.46:0/512705936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d2807a760 con 0x7f9d281065f0 2024-04-27T03:55:07.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.053+0000 7f9d1f7fe700 1 -- 172.21.15.46:0/512705936 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f9d1001ab80 con 0x7f9d281065f0 2024-04-27T03:55:07.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.053+0000 7f9d2cdef700 1 -- 172.21.15.46:0/512705936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d28062510 con 0x7f9d281065f0 2024-04-27T03:55:07.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.053+0000 7f9d1f7fe700 1 --2- 172.21.15.46:0/512705936 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f9d14038720 0x7f9d1403abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:07.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.053+0000 7f9d1f7fe700 1 -- 172.21.15.46:0/512705936 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9d1004c990 con 0x7f9d281065f0 2024-04-27T03:55:07.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.053+0000 7f9d25d9b700 1 --2- 172.21.15.46:0/512705936 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f9d14038720 0x7f9d1403abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:07.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.054+0000 7f9d25d9b700 1 --2- 172.21.15.46:0/512705936 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f9d14038720 0x7f9d1403abe0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9d18006fd0 tx=0x7f9d18006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:07.518 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.058+0000 7f9d1f7fe700 1 -- 172.21.15.46:0/512705936 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f9d1001a430 con 0x7f9d281065f0 2024-04-27T03:55:07.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.317+0000 7f9d2cdef700 1 -- 172.21.15.46:0/512705936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f9d2807aca0 con 0x7f9d281065f0 2024-04-27T03:55:07.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.898+0000 7f9d1f7fe700 1 -- 172.21.15.46:0/512705936 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 9) v1 ==== 45569+0+0 (secure 0 0 0) 0x7f9d1002d770 con 0x7f9d281065f0 2024-04-27T03:55:07.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.900+0000 7f9d1f7fe700 1 -- 172.21.15.46:0/512705936 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f9d1004c2e0 con 0x7f9d281065f0 2024-04-27T03:55:07.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.903+0000 7f9d2cdef700 1 -- 172.21.15.46:0/512705936 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f9d14038720 msgr2=0x7f9d1403abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:07.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.903+0000 7f9d2cdef700 1 --2- 172.21.15.46:0/512705936 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f9d14038720 0x7f9d1403abe0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9d18006fd0 tx=0x7f9d18006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:07.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.903+0000 7f9d2cdef700 1 -- 172.21.15.46:0/512705936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d281065f0 msgr2=0x7f9d2807baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:07.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.903+0000 7f9d2cdef700 1 --2- 172.21.15.46:0/512705936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d281065f0 0x7f9d2807baf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9d100117b0 tx=0x7f9d10011890 comp rx=0 tx=0).stop 2024-04-27T03:55:07.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.903+0000 7f9d2cdef700 1 -- 172.21.15.46:0/512705936 shutdown_connections 2024-04-27T03:55:07.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.903+0000 7f9d2cdef700 1 --2- 172.21.15.46:0/512705936 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f9d14038720 0x7f9d1403abe0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:07.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.903+0000 7f9d2cdef700 1 --2- 172.21.15.46:0/512705936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d281065f0 0x7f9d2807baf0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:07.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.903+0000 7f9d2cdef700 1 -- 172.21.15.46:0/512705936 >> 172.21.15.46:0/512705936 conn(0x7f9d28101bd0 msgr2=0x7f9d28102780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:07.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.903+0000 7f9d2cdef700 1 -- 172.21.15.46:0/512705936 shutdown_connections 2024-04-27T03:55:07.519 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:06.903+0000 7f9d2cdef700 1 -- 172.21.15.46:0/512705936 wait complete. 2024-04-27T03:55:08.159 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:07 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/512705936' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-27T03:55:08.159 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:07 smithi046 ceph-mon[26899]: mgrmap e9: smithi046.qrfdvr(active, since 26s) 2024-04-27T03:55:08.843 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-04-27T03:55:08.843 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-27T03:55:08.844 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-27T03:55:08.844 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "active_name": "smithi046.qrfdvr", 2024-04-27T03:55:08.844 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-27T03:55:08.844 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-04-27T03:55:08.844 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.950+0000 7f786150b700 1 Processor -- start 2024-04-27T03:55:08.844 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.950+0000 7f786150b700 1 -- start start 2024-04-27T03:55:08.845 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.951+0000 7f786150b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c1054e0 0x7f785c105900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:08.845 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.951+0000 7f786150b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c105ed0 con 0x7f785c1054e0 2024-04-27T03:55:08.845 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.951+0000 7f785affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c1054e0 0x7f785c105900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:08.845 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.951+0000 7f785affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c1054e0 0x7f785c105900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55338/0 (socket says 172.21.15.46:55338) 2024-04-27T03:55:08.845 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.951+0000 7f785affd700 1 -- 172.21.15.46:0/1225065967 learned_addr learned my addr 172.21.15.46:0/1225065967 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:08.845 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.951+0000 7f785affd700 1 -- 172.21.15.46:0/1225065967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c1066e0 con 0x7f785c1054e0 2024-04-27T03:55:08.846 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.951+0000 7f785affd700 1 --2- 172.21.15.46:0/1225065967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c1054e0 0x7f785c105900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f784c0098a0 tx=0x7f784c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58236fd59c79f538 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:08.846 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.951+0000 7f7859ffb700 1 -- 172.21.15.46:0/1225065967 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f784c004180 con 0x7f785c1054e0 2024-04-27T03:55:08.846 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.951+0000 7f7859ffb700 1 -- 172.21.15.46:0/1225065967 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f784c0042e0 con 0x7f785c1054e0 2024-04-27T03:55:08.846 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.951+0000 7f7859ffb700 1 -- 172.21.15.46:0/1225065967 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f784c011b10 con 0x7f785c1054e0 2024-04-27T03:55:08.846 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.952+0000 7f786150b700 1 -- 172.21.15.46:0/1225065967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c1054e0 msgr2=0x7f785c105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:08.846 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.952+0000 7f786150b700 1 --2- 172.21.15.46:0/1225065967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c1054e0 0x7f785c105900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f784c0098a0 tx=0x7f784c009bb0 comp rx=0 tx=0).stop 2024-04-27T03:55:08.847 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.952+0000 7f786150b700 1 -- 172.21.15.46:0/1225065967 shutdown_connections 2024-04-27T03:55:08.847 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.952+0000 7f786150b700 1 --2- 172.21.15.46:0/1225065967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c1054e0 0x7f785c105900 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:08.847 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.952+0000 7f786150b700 1 -- 172.21.15.46:0/1225065967 >> 172.21.15.46:0/1225065967 conn(0x7f785c100b00 msgr2=0x7f785c102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:08.847 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.952+0000 7f786150b700 1 -- 172.21.15.46:0/1225065967 shutdown_connections 2024-04-27T03:55:08.847 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.952+0000 7f786150b700 1 -- 172.21.15.46:0/1225065967 wait complete. 2024-04-27T03:55:08.847 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.953+0000 7f786150b700 1 Processor -- start 2024-04-27T03:55:08.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.953+0000 7f786150b700 1 -- start start 2024-04-27T03:55:08.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.953+0000 7f786150b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c1054e0 0x7f785c1905a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:08.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.953+0000 7f786150b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c190ae0 con 0x7f785c1054e0 2024-04-27T03:55:08.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.953+0000 7f785affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c1054e0 0x7f785c1905a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:08.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.953+0000 7f785affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c1054e0 0x7f785c1905a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55346/0 (socket says 172.21.15.46:55346) 2024-04-27T03:55:08.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.953+0000 7f785affd700 1 -- 172.21.15.46:0/2039249663 learned_addr learned my addr 172.21.15.46:0/2039249663 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:08.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.953+0000 7f785affd700 1 -- 172.21.15.46:0/2039249663 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784c009550 con 0x7f785c1054e0 2024-04-27T03:55:08.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.954+0000 7f785affd700 1 --2- 172.21.15.46:0/2039249663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c1054e0 0x7f785c1905a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f784c0117b0 tx=0x7f784c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:08.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.954+0000 7f7853fff700 1 -- 172.21.15.46:0/2039249663 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f784c0038b0 con 0x7f785c1054e0 2024-04-27T03:55:08.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.954+0000 7f7853fff700 1 -- 172.21.15.46:0/2039249663 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f784c003a10 con 0x7f785c1054e0 2024-04-27T03:55:08.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.954+0000 7f7853fff700 1 -- 172.21.15.46:0/2039249663 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f784c0246e0 con 0x7f785c1054e0 2024-04-27T03:55:08.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.954+0000 7f786150b700 1 -- 172.21.15.46:0/2039249663 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f785c18ed20 con 0x7f785c1054e0 2024-04-27T03:55:08.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.954+0000 7f786150b700 1 -- 172.21.15.46:0/2039249663 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f785c18f100 con 0x7f785c1054e0 2024-04-27T03:55:08.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.954+0000 7f7853fff700 1 -- 172.21.15.46:0/2039249663 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 9) v1 ==== 45569+0+0 (secure 0 0 0) 0x7f784c02d3f0 con 0x7f785c1054e0 2024-04-27T03:55:08.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.955+0000 7f7853fff700 1 --2- 172.21.15.46:0/2039249663 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f78480387c0 0x7f784803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:08.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.955+0000 7f7853fff700 1 -- 172.21.15.46:0/2039249663 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f784c04c930 con 0x7f785c1054e0 2024-04-27T03:55:08.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.955+0000 7f785a7fc700 1 -- 172.21.15.46:0/2039249663 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f78480387c0 msgr2=0x7f784803ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/333567692 2024-04-27T03:55:08.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.955+0000 7f785a7fc700 1 --2- 172.21.15.46:0/2039249663 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f78480387c0 0x7f784803ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-27T03:55:08.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.955+0000 7f786150b700 1 -- 172.21.15.46:0/2039249663 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f785c18fb70 con 0x7f785c1054e0 2024-04-27T03:55:08.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:07.960+0000 7f7853fff700 1 -- 172.21.15.46:0/2039249663 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f784c02d740 con 0x7f785c1054e0 2024-04-27T03:55:08.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:08.156+0000 7f785a7fc700 1 -- 172.21.15.46:0/2039249663 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f78480387c0 msgr2=0x7f784803ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/333567692 2024-04-27T03:55:08.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:08.156+0000 7f785a7fc700 1 --2- 172.21.15.46:0/2039249663 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f78480387c0 0x7f784803ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-27T03:55:08.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:08.254+0000 7f786150b700 1 -- 172.21.15.46:0/2039249663 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f785c193fa0 con 0x7f785c1054e0 2024-04-27T03:55:08.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:08.255+0000 7f7853fff700 1 -- 172.21.15.46:0/2039249663 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f784c0107f0 con 0x7f785c1054e0 2024-04-27T03:55:08.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:08.258+0000 7f786150b700 1 -- 172.21.15.46:0/2039249663 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f78480387c0 msgr2=0x7f784803ac80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T03:55:08.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:08.258+0000 7f786150b700 1 --2- 172.21.15.46:0/2039249663 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f78480387c0 0x7f784803ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:08.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:08.258+0000 7f786150b700 1 -- 172.21.15.46:0/2039249663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c1054e0 msgr2=0x7f785c1905a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:08.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:08.258+0000 7f786150b700 1 --2- 172.21.15.46:0/2039249663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c1054e0 0x7f785c1905a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f784c0117b0 tx=0x7f784c011890 comp rx=0 tx=0).stop 2024-04-27T03:55:08.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:08.259+0000 7f786150b700 1 -- 172.21.15.46:0/2039249663 shutdown_connections 2024-04-27T03:55:08.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:08.259+0000 7f786150b700 1 --2- 172.21.15.46:0/2039249663 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7f78480387c0 0x7f784803ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:08.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:08.259+0000 7f786150b700 1 --2- 172.21.15.46:0/2039249663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c1054e0 0x7f785c1905a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:08.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:08.259+0000 7f786150b700 1 -- 172.21.15.46:0/2039249663 >> 172.21.15.46:0/2039249663 conn(0x7f785c100b00 msgr2=0x7f785c1019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:08.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:08.260+0000 7f786150b700 1 -- 172.21.15.46:0/2039249663 shutdown_connections 2024-04-27T03:55:08.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:08.260+0000 7f786150b700 1 -- 172.21.15.46:0/2039249663 wait complete. 2024-04-27T03:55:08.852 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for the mgr to restart... 2024-04-27T03:55:08.852 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr epoch 9... 2024-04-27T03:55:09.081 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:08 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2039249663' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-27T03:55:14.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:14 smithi046 ceph-mon[26899]: Active manager daemon smithi046.qrfdvr restarted 2024-04-27T03:55:14.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:14 smithi046 ceph-mon[26899]: Activating manager daemon smithi046.qrfdvr 2024-04-27T03:55:14.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:14 smithi046 ceph-mon[26899]: osdmap e3: 0 total, 0 up, 0 in 2024-04-27T03:55:14.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:14 smithi046 ceph-mon[26899]: mgrmap e10: smithi046.qrfdvr(active, starting, since 0.00329504s) 2024-04-27T03:55:14.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:14 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-04-27T03:55:14.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:14 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mgr metadata", "who": "smithi046.qrfdvr", "id": "smithi046.qrfdvr"}]: dispatch 2024-04-27T03:55:14.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:14 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T03:55:14.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:14 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T03:55:14.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:14 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T03:55:14.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:14 smithi046 ceph-mon[26899]: Manager daemon smithi046.qrfdvr is now available 2024-04-27T03:55:14.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:14 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:55:14.428 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:14 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.qrfdvr/mirror_snapshot_schedule"}]: dispatch 2024-04-27T03:55:15.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:15 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.qrfdvr/trash_purge_schedule"}]: dispatch 2024-04-27T03:55:15.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:15 smithi046 ceph-mon[26899]: mgrmap e11: smithi046.qrfdvr(active, since 1.00607s) 2024-04-27T03:55:15.781 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-04-27T03:55:15.781 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-27T03:55:15.781 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-27T03:55:15.781 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-04-27T03:55:15.781 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.464+0000 7fa84c34f700 1 Processor -- start 2024-04-27T03:55:15.781 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.464+0000 7fa84c34f700 1 -- start start 2024-04-27T03:55:15.781 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.465+0000 7fa84c34f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa844104e80 0x7fa8441052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:15.781 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.465+0000 7fa84c34f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa844079e50 con 0x7fa844104e80 2024-04-27T03:55:15.781 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.465+0000 7fa84a0eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa844104e80 0x7fa8441052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:15.781 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.465+0000 7fa84a0eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa844104e80 0x7fa8441052a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55354/0 (socket says 172.21.15.46:55354) 2024-04-27T03:55:15.782 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.465+0000 7fa84a0eb700 1 -- 172.21.15.46:0/3983585930 learned_addr learned my addr 172.21.15.46:0/3983585930 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:15.782 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.465+0000 7fa84a0eb700 1 -- 172.21.15.46:0/3983585930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8441057e0 con 0x7fa844104e80 2024-04-27T03:55:15.782 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.465+0000 7fa84a0eb700 1 --2- 172.21.15.46:0/3983585930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa844104e80 0x7fa8441052a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa8400098a0 tx=0x7fa840009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce3a9e4d9bfaa7b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:15.782 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.465+0000 7fa8490e9700 1 -- 172.21.15.46:0/3983585930 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa840004180 con 0x7fa844104e80 2024-04-27T03:55:15.782 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.465+0000 7fa8490e9700 1 -- 172.21.15.46:0/3983585930 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8400042e0 con 0x7fa844104e80 2024-04-27T03:55:15.782 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.465+0000 7fa8490e9700 1 -- 172.21.15.46:0/3983585930 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa840011b10 con 0x7fa844104e80 2024-04-27T03:55:15.782 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.466+0000 7fa84c34f700 1 -- 172.21.15.46:0/3983585930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa844104e80 msgr2=0x7fa8441052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:15.782 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.466+0000 7fa84c34f700 1 --2- 172.21.15.46:0/3983585930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa844104e80 0x7fa8441052a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa8400098a0 tx=0x7fa840009bb0 comp rx=0 tx=0).stop 2024-04-27T03:55:15.782 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.466+0000 7fa84c34f700 1 -- 172.21.15.46:0/3983585930 shutdown_connections 2024-04-27T03:55:15.782 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.466+0000 7fa84c34f700 1 --2- 172.21.15.46:0/3983585930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa844104e80 0x7fa8441052a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:15.782 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.466+0000 7fa84c34f700 1 -- 172.21.15.46:0/3983585930 >> 172.21.15.46:0/3983585930 conn(0x7fa844100aa0 msgr2=0x7fa844102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:15.782 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.466+0000 7fa84c34f700 1 -- 172.21.15.46:0/3983585930 shutdown_connections 2024-04-27T03:55:15.783 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.466+0000 7fa84c34f700 1 -- 172.21.15.46:0/3983585930 wait complete. 2024-04-27T03:55:15.783 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.467+0000 7fa84c34f700 1 Processor -- start 2024-04-27T03:55:15.783 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.467+0000 7fa84c34f700 1 -- start start 2024-04-27T03:55:15.783 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.467+0000 7fa84c34f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa844104e80 0x7fa84419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:15.783 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.467+0000 7fa84c34f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84419ca40 con 0x7fa844104e80 2024-04-27T03:55:15.783 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.467+0000 7fa84a0eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa844104e80 0x7fa84419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:15.783 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.467+0000 7fa84a0eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa844104e80 0x7fa84419c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55366/0 (socket says 172.21.15.46:55366) 2024-04-27T03:55:15.783 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.467+0000 7fa84a0eb700 1 -- 172.21.15.46:0/1852065313 learned_addr learned my addr 172.21.15.46:0/1852065313 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:15.783 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.467+0000 7fa84a0eb700 1 -- 172.21.15.46:0/1852065313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa840009550 con 0x7fa844104e80 2024-04-27T03:55:15.783 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.468+0000 7fa84a0eb700 1 --2- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa844104e80 0x7fa84419c500 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa8400117b0 tx=0x7fa840011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:15.783 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.468+0000 7fa8377fe700 1 -- 172.21.15.46:0/1852065313 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8400038b0 con 0x7fa844104e80 2024-04-27T03:55:15.783 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.468+0000 7fa8377fe700 1 -- 172.21.15.46:0/1852065313 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa840003a10 con 0x7fa844104e80 2024-04-27T03:55:15.783 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.468+0000 7fa84c34f700 1 -- 172.21.15.46:0/1852065313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84419cc40 con 0x7fa844104e80 2024-04-27T03:55:15.783 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.468+0000 7fa8377fe700 1 -- 172.21.15.46:0/1852065313 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa840024780 con 0x7fa844104e80 2024-04-27T03:55:15.784 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.468+0000 7fa84c34f700 1 -- 172.21.15.46:0/1852065313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84419d0e0 con 0x7fa844104e80 2024-04-27T03:55:15.784 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.469+0000 7fa8377fe700 1 -- 172.21.15.46:0/1852065313 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 9) v1 ==== 45569+0+0 (secure 0 0 0) 0x7fa8400248e0 con 0x7fa844104e80 2024-04-27T03:55:15.784 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.469+0000 7fa8377fe700 1 --2- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa8300387c0 0x7fa83003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:15.784 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.469+0000 7fa84c34f700 1 -- 172.21.15.46:0/1852065313 --> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa844106640 con 0x7fa8300387c0 2024-04-27T03:55:15.784 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.469+0000 7fa8377fe700 1 -- 172.21.15.46:0/1852065313 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa84004c970 con 0x7fa844104e80 2024-04-27T03:55:15.784 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.469+0000 7fa8498ea700 1 -- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa8300387c0 msgr2=0x7fa83003ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/333567692 2024-04-27T03:55:15.784 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.469+0000 7fa8498ea700 1 --2- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa8300387c0 0x7fa83003ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-27T03:55:15.784 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.670+0000 7fa8498ea700 1 -- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa8300387c0 msgr2=0x7fa83003ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/333567692 2024-04-27T03:55:15.784 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:09.670+0000 7fa8498ea700 1 --2- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa8300387c0 0x7fa83003ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-27T03:55:15.784 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:10.071+0000 7fa8498ea700 1 -- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa8300387c0 msgr2=0x7fa83003ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/333567692 2024-04-27T03:55:15.784 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:10.071+0000 7fa8498ea700 1 --2- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa8300387c0 0x7fa83003ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-27T03:55:15.784 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:10.872+0000 7fa8498ea700 1 -- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa8300387c0 msgr2=0x7fa83003ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/333567692 2024-04-27T03:55:15.784 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:10.872+0000 7fa8498ea700 1 --2- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa8300387c0 0x7fa83003ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-27T03:55:15.785 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:12.473+0000 7fa8498ea700 1 -- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa8300387c0 msgr2=0x7fa83003ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/333567692 2024-04-27T03:55:15.785 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:12.473+0000 7fa8498ea700 1 --2- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa8300387c0 0x7fa83003ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-27T03:55:15.785 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:14.072+0000 7fa8377fe700 1 -- 172.21.15.46:0/1852065313 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mgrmap(e 10) v1 ==== 45336+0+0 (secure 0 0 0) 0x7fa84001ac10 con 0x7fa844104e80 2024-04-27T03:55:15.785 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:14.072+0000 7fa8377fe700 1 -- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa8300387c0 msgr2=0x7fa83003ac80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T03:55:15.785 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:14.072+0000 7fa8377fe700 1 --2- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa8300387c0 0x7fa83003ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:15.785 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.074+0000 7fa8377fe700 1 -- 172.21.15.46:0/1852065313 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 11) v1 ==== 45463+0+0 (secure 0 0 0) 0x7fa84004e300 con 0x7fa844104e80 2024-04-27T03:55:15.785 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.075+0000 7fa8377fe700 1 --2- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fa83003bfe0 0x7fa83003e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:15.785 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.075+0000 7fa8377fe700 1 -- 172.21.15.46:0/1852065313 --> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa844106640 con 0x7fa83003bfe0 2024-04-27T03:55:15.785 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.075+0000 7fa8498ea700 1 --2- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fa83003bfe0 0x7fa83003e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:15.786 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.075+0000 7fa8498ea700 1 --2- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fa83003bfe0 0x7fa83003e3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa838003a10 tx=0x7fa838009270 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:15.786 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.076+0000 7fa8377fe700 1 -- 172.21.15.46:0/1852065313 <== mgr.14162 v2:172.21.15.46:6800/3280787672 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fa844106640 con 0x7fa83003bfe0 2024-04-27T03:55:15.786 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.082+0000 7fa84c34f700 1 -- 172.21.15.46:0/1852065313 --> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa8441067b0 con 0x7fa83003bfe0 2024-04-27T03:55:15.786 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.083+0000 7fa8377fe700 1 -- 172.21.15.46:0/1852065313 <== mgr.14162 v2:172.21.15.46:6800/3280787672 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fa8441067b0 con 0x7fa83003bfe0 2024-04-27T03:55:15.786 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.083+0000 7fa84c34f700 1 -- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fa83003bfe0 msgr2=0x7fa83003e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:15.786 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.083+0000 7fa84c34f700 1 --2- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fa83003bfe0 0x7fa83003e3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa838003a10 tx=0x7fa838009270 comp rx=0 tx=0).stop 2024-04-27T03:55:15.786 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.083+0000 7fa84c34f700 1 -- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa844104e80 msgr2=0x7fa84419c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:15.786 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.083+0000 7fa84c34f700 1 --2- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa844104e80 0x7fa84419c500 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa8400117b0 tx=0x7fa840011890 comp rx=0 tx=0).stop 2024-04-27T03:55:15.786 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.083+0000 7fa84c34f700 1 -- 172.21.15.46:0/1852065313 shutdown_connections 2024-04-27T03:55:15.786 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.083+0000 7fa84c34f700 1 --2- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fa83003bfe0 0x7fa83003e3d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:15.786 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.083+0000 7fa84c34f700 1 --2- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa844104e80 0x7fa84419c500 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:15.786 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.083+0000 7fa84c34f700 1 --2- 172.21.15.46:0/1852065313 >> [v2:172.21.15.46:6800/333567692,v1:172.21.15.46:6801/333567692] conn(0x7fa8300387c0 0x7fa83003ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:15.786 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.083+0000 7fa84c34f700 1 -- 172.21.15.46:0/1852065313 >> 172.21.15.46:0/1852065313 conn(0x7fa844100aa0 msgr2=0x7fa844193390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:15.786 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.083+0000 7fa84c34f700 1 -- 172.21.15.46:0/1852065313 shutdown_connections 2024-04-27T03:55:15.787 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:15.083+0000 7fa84c34f700 1 -- 172.21.15.46:0/1852065313 wait complete. 2024-04-27T03:55:15.787 INFO:teuthology.orchestra.run.smithi046.stdout:mgr epoch 9 is available 2024-04-27T03:55:15.787 INFO:teuthology.orchestra.run.smithi046.stdout:Generating a dashboard self-signed certificate... 2024-04-27T03:55:16.668 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:16 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:16.668 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:16 smithi046 ceph-mon[26899]: [27/Apr/2024:03:55:15] ENGINE Bus STARTING 2024-04-27T03:55:16.668 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:16 smithi046 ceph-mon[26899]: [27/Apr/2024:03:55:15] ENGINE Serving on https://172.21.15.46:7150 2024-04-27T03:55:16.669 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:16 smithi046 ceph-mon[26899]: [27/Apr/2024:03:55:15] ENGINE Serving on http://172.21.15.46:8765 2024-04-27T03:55:16.669 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:16 smithi046 ceph-mon[26899]: [27/Apr/2024:03:55:15] ENGINE Bus STARTED 2024-04-27T03:55:16.669 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:16 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:17.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-27T03:55:17.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.267+0000 7f4d2ee9f700 1 Processor -- start 2024-04-27T03:55:17.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.267+0000 7f4d2ee9f700 1 -- start start 2024-04-27T03:55:17.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.267+0000 7f4d2ee9f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d2807be50 0x7f4d2807a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:17.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.267+0000 7f4d2ee9f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2807a840 con 0x7f4d2807be50 2024-04-27T03:55:17.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.267+0000 7f4d2cc3b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d2807be50 0x7f4d2807a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:17.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.267+0000 7f4d2cc3b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d2807be50 0x7f4d2807a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37544/0 (socket says 172.21.15.46:37544) 2024-04-27T03:55:17.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.267+0000 7f4d2cc3b700 1 -- 172.21.15.46:0/3940690858 learned_addr learned my addr 172.21.15.46:0/3940690858 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:17.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.268+0000 7f4d2cc3b700 1 -- 172.21.15.46:0/3940690858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2807a980 con 0x7f4d2807be50 2024-04-27T03:55:17.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.268+0000 7f4d2cc3b700 1 --2- 172.21.15.46:0/3940690858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d2807be50 0x7f4d2807a300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c0098a0 tx=0x7f4d1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=559f8babe498c225 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:17.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.268+0000 7f4d277fe700 1 -- 172.21.15.46:0/3940690858 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d1c004180 con 0x7f4d2807be50 2024-04-27T03:55:17.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.268+0000 7f4d277fe700 1 -- 172.21.15.46:0/3940690858 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d1c0042e0 con 0x7f4d2807be50 2024-04-27T03:55:17.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.268+0000 7f4d277fe700 1 -- 172.21.15.46:0/3940690858 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d1c011b10 con 0x7f4d2807be50 2024-04-27T03:55:17.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.269+0000 7f4d2ee9f700 1 -- 172.21.15.46:0/3940690858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d2807be50 msgr2=0x7f4d2807a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:17.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.269+0000 7f4d2ee9f700 1 --2- 172.21.15.46:0/3940690858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d2807be50 0x7f4d2807a300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c0098a0 tx=0x7f4d1c009bb0 comp rx=0 tx=0).stop 2024-04-27T03:55:17.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.269+0000 7f4d2ee9f700 1 -- 172.21.15.46:0/3940690858 shutdown_connections 2024-04-27T03:55:17.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.269+0000 7f4d2ee9f700 1 --2- 172.21.15.46:0/3940690858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d2807be50 0x7f4d2807a300 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:17.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.269+0000 7f4d2ee9f700 1 -- 172.21.15.46:0/3940690858 >> 172.21.15.46:0/3940690858 conn(0x7f4d28101b50 msgr2=0x7f4d28103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:17.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.269+0000 7f4d2ee9f700 1 -- 172.21.15.46:0/3940690858 shutdown_connections 2024-04-27T03:55:17.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.269+0000 7f4d2ee9f700 1 -- 172.21.15.46:0/3940690858 wait complete. 2024-04-27T03:55:17.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.269+0000 7f4d2ee9f700 1 Processor -- start 2024-04-27T03:55:17.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.270+0000 7f4d2ee9f700 1 -- start start 2024-04-27T03:55:17.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.270+0000 7f4d2ee9f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d2807be50 0x7f4d2819c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:17.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.270+0000 7f4d2ee9f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2819ca60 con 0x7f4d2807be50 2024-04-27T03:55:17.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.270+0000 7f4d2cc3b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d2807be50 0x7f4d2819c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:17.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.270+0000 7f4d2cc3b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d2807be50 0x7f4d2819c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37548/0 (socket says 172.21.15.46:37548) 2024-04-27T03:55:17.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.270+0000 7f4d2cc3b700 1 -- 172.21.15.46:0/206227370 learned_addr learned my addr 172.21.15.46:0/206227370 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:17.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.270+0000 7f4d2cc3b700 1 -- 172.21.15.46:0/206227370 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c009550 con 0x7f4d2807be50 2024-04-27T03:55:17.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.270+0000 7f4d2cc3b700 1 --2- 172.21.15.46:0/206227370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d2807be50 0x7f4d2819c520 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c009fb0 tx=0x7f4d1c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:17.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.270+0000 7f4d25ffb700 1 -- 172.21.15.46:0/206227370 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d1c0038b0 con 0x7f4d2807be50 2024-04-27T03:55:17.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.271+0000 7f4d2ee9f700 1 -- 172.21.15.46:0/206227370 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2819cc60 con 0x7f4d2807be50 2024-04-27T03:55:17.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.271+0000 7f4d25ffb700 1 -- 172.21.15.46:0/206227370 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d1c003a10 con 0x7f4d2807be50 2024-04-27T03:55:17.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.271+0000 7f4d25ffb700 1 -- 172.21.15.46:0/206227370 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d1c019750 con 0x7f4d2807be50 2024-04-27T03:55:17.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.271+0000 7f4d2ee9f700 1 -- 172.21.15.46:0/206227370 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2819d100 con 0x7f4d2807be50 2024-04-27T03:55:17.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.272+0000 7f4d25ffb700 1 -- 172.21.15.46:0/206227370 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 11) v1 ==== 45463+0+0 (secure 0 0 0) 0x7f4d1c0199c0 con 0x7f4d2807be50 2024-04-27T03:55:17.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.272+0000 7f4d25ffb700 1 --2- 172.21.15.46:0/206227370 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f4d18038610 0x7f4d1803aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:17.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.272+0000 7f4d25ffb700 1 -- 172.21.15.46:0/206227370 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4d1c04d830 con 0x7f4d2807be50 2024-04-27T03:55:17.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.272+0000 7f4d2ee9f700 1 -- 172.21.15.46:0/206227370 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d2810c8f0 con 0x7f4d2807be50 2024-04-27T03:55:17.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.272+0000 7f4d27fff700 1 --2- 172.21.15.46:0/206227370 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f4d18038610 0x7f4d1803aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:17.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.273+0000 7f4d27fff700 1 --2- 172.21.15.46:0/206227370 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f4d18038610 0x7f4d1803aad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4d14006fd0 tx=0x7f4d14006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:17.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.277+0000 7f4d25ffb700 1 -- 172.21.15.46:0/206227370 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f4d1c019d10 con 0x7f4d2807be50 2024-04-27T03:55:17.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.429+0000 7f4d25ffb700 1 -- 172.21.15.46:0/206227370 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 12) v1 ==== 45569+0+0 (secure 0 0 0) 0x7f4d1c01ac10 con 0x7f4d2807be50 2024-04-27T03:55:17.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.490+0000 7f4d2ee9f700 1 -- 172.21.15.46:0/206227370 --> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d2819f990 con 0x7f4d18038610 2024-04-27T03:55:17.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.665+0000 7f4d25ffb700 1 -- 172.21.15.46:0/206227370 <== mgr.14162 v2:172.21.15.46:6800/3280787672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f4d2819f990 con 0x7f4d18038610 2024-04-27T03:55:17.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.668+0000 7f4d2ee9f700 1 -- 172.21.15.46:0/206227370 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f4d18038610 msgr2=0x7f4d1803aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:17.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.668+0000 7f4d2ee9f700 1 --2- 172.21.15.46:0/206227370 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f4d18038610 0x7f4d1803aad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4d14006fd0 tx=0x7f4d14006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:17.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.668+0000 7f4d2ee9f700 1 -- 172.21.15.46:0/206227370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d2807be50 msgr2=0x7f4d2819c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:17.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.668+0000 7f4d2ee9f700 1 --2- 172.21.15.46:0/206227370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d2807be50 0x7f4d2819c520 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c009fb0 tx=0x7f4d1c011890 comp rx=0 tx=0).stop 2024-04-27T03:55:17.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.668+0000 7f4d2ee9f700 1 -- 172.21.15.46:0/206227370 shutdown_connections 2024-04-27T03:55:17.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.668+0000 7f4d2ee9f700 1 --2- 172.21.15.46:0/206227370 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f4d18038610 0x7f4d1803aad0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:17.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.668+0000 7f4d2ee9f700 1 --2- 172.21.15.46:0/206227370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d2807be50 0x7f4d2819c520 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:17.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.668+0000 7f4d2ee9f700 1 -- 172.21.15.46:0/206227370 >> 172.21.15.46:0/206227370 conn(0x7f4d28101b50 msgr2=0x7f4d28102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:17.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.668+0000 7f4d2ee9f700 1 -- 172.21.15.46:0/206227370 shutdown_connections 2024-04-27T03:55:17.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:16.668+0000 7f4d2ee9f700 1 -- 172.21.15.46:0/206227370 wait complete. 2024-04-27T03:55:17.247 INFO:teuthology.orchestra.run.smithi046.stdout:Creating initial admin user... 2024-04-27T03:55:17.564 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:17 smithi046 ceph-mon[26899]: mgrmap e12: smithi046.qrfdvr(active, since 2s) 2024-04-27T03:55:17.564 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:17 smithi046 ceph-mon[26899]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:55:17.564 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:17 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:17.564 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:17 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:18.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:18 smithi046 ceph-mon[26899]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:55:18.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:18 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:18.981 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$DTvs3/sCQW.vOPM6AL3sLOn2qQclSTY/6kZvAIV9XHDuRc2W9Y6lK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714190118, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-27T03:55:18.982 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.669+0000 7f91256d5700 1 Processor -- start 2024-04-27T03:55:18.982 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.669+0000 7f91256d5700 1 -- start start 2024-04-27T03:55:18.982 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.669+0000 7f91256d5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91201065c0 0x7f91201069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:18.982 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.669+0000 7f91256d5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9120106fb0 con 0x7f91201065c0 2024-04-27T03:55:18.982 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.669+0000 7f911effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91201065c0 0x7f91201069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:18.982 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.669+0000 7f911effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91201065c0 0x7f91201069e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37556/0 (socket says 172.21.15.46:37556) 2024-04-27T03:55:18.982 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.669+0000 7f911effd700 1 -- 172.21.15.46:0/1975455847 learned_addr learned my addr 172.21.15.46:0/1975455847 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:18.982 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.669+0000 7f911effd700 1 -- 172.21.15.46:0/1975455847 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91201077c0 con 0x7f91201065c0 2024-04-27T03:55:18.982 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.670+0000 7f911effd700 1 --2- 172.21.15.46:0/1975455847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91201065c0 0x7f91201069e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f91100098a0 tx=0x7f9110009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee049ab9a8856c67 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:18.982 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.670+0000 7f911dffb700 1 -- 172.21.15.46:0/1975455847 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9110004180 con 0x7f91201065c0 2024-04-27T03:55:18.982 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.670+0000 7f911dffb700 1 -- 172.21.15.46:0/1975455847 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f91100042e0 con 0x7f91201065c0 2024-04-27T03:55:18.983 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.670+0000 7f911dffb700 1 -- 172.21.15.46:0/1975455847 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9110011b10 con 0x7f91201065c0 2024-04-27T03:55:18.983 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.670+0000 7f91256d5700 1 -- 172.21.15.46:0/1975455847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91201065c0 msgr2=0x7f91201069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:18.983 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.670+0000 7f91256d5700 1 --2- 172.21.15.46:0/1975455847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91201065c0 0x7f91201069e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f91100098a0 tx=0x7f9110009bb0 comp rx=0 tx=0).stop 2024-04-27T03:55:18.983 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.670+0000 7f91256d5700 1 -- 172.21.15.46:0/1975455847 shutdown_connections 2024-04-27T03:55:18.983 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.670+0000 7f91256d5700 1 --2- 172.21.15.46:0/1975455847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91201065c0 0x7f91201069e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:18.983 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.670+0000 7f91256d5700 1 -- 172.21.15.46:0/1975455847 >> 172.21.15.46:0/1975455847 conn(0x7f9120101be0 msgr2=0x7f9120104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:18.983 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.670+0000 7f91256d5700 1 -- 172.21.15.46:0/1975455847 shutdown_connections 2024-04-27T03:55:18.983 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.670+0000 7f91256d5700 1 -- 172.21.15.46:0/1975455847 wait complete. 2024-04-27T03:55:18.983 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.671+0000 7f91256d5700 1 Processor -- start 2024-04-27T03:55:18.983 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.671+0000 7f91256d5700 1 -- start start 2024-04-27T03:55:18.983 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.671+0000 7f91256d5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91201065c0 0x7f912019a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:18.983 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.671+0000 7f91256d5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912019a890 con 0x7f91201065c0 2024-04-27T03:55:18.983 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.671+0000 7f911effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91201065c0 0x7f912019a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:18.984 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.672+0000 7f911effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91201065c0 0x7f912019a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37568/0 (socket says 172.21.15.46:37568) 2024-04-27T03:55:18.984 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.672+0000 7f911effd700 1 -- 172.21.15.46:0/522454049 learned_addr learned my addr 172.21.15.46:0/522454049 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:18.984 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.672+0000 7f911effd700 1 -- 172.21.15.46:0/522454049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9110009550 con 0x7f91201065c0 2024-04-27T03:55:18.984 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.672+0000 7f911effd700 1 --2- 172.21.15.46:0/522454049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91201065c0 0x7f912019a350 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9110009fb0 tx=0x7f9110011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:18.984 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.672+0000 7f9107fff700 1 -- 172.21.15.46:0/522454049 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91100038b0 con 0x7f91201065c0 2024-04-27T03:55:18.984 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.672+0000 7f9107fff700 1 -- 172.21.15.46:0/522454049 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9110003a10 con 0x7f91201065c0 2024-04-27T03:55:18.984 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.672+0000 7f91256d5700 1 -- 172.21.15.46:0/522454049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f912019aa90 con 0x7f91201065c0 2024-04-27T03:55:18.984 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.672+0000 7f9107fff700 1 -- 172.21.15.46:0/522454049 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9110019770 con 0x7f91201065c0 2024-04-27T03:55:18.984 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.672+0000 7f91256d5700 1 -- 172.21.15.46:0/522454049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f912019af30 con 0x7f91201065c0 2024-04-27T03:55:18.984 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.673+0000 7f9107fff700 1 -- 172.21.15.46:0/522454049 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) v1 ==== 45569+0+0 (secure 0 0 0) 0x7f911001a430 con 0x7f91201065c0 2024-04-27T03:55:18.984 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.673+0000 7f91256d5700 1 -- 172.21.15.46:0/522454049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912010a9f0 con 0x7f91201065c0 2024-04-27T03:55:18.984 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.673+0000 7f9107fff700 1 --2- 172.21.15.46:0/522454049 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f9108038780 0x7f910803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:18.984 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.673+0000 7f9107fff700 1 -- 172.21.15.46:0/522454049 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f911004c8e0 con 0x7f91201065c0 2024-04-27T03:55:18.984 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.673+0000 7f911e7fc700 1 --2- 172.21.15.46:0/522454049 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f9108038780 0x7f910803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:18.985 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.674+0000 7f911e7fc700 1 --2- 172.21.15.46:0/522454049 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f9108038780 0x7f910803ac40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9114006fd0 tx=0x7f9114006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:18.985 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.678+0000 7f9107fff700 1 -- 172.21.15.46:0/522454049 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f9110019df0 con 0x7f91201065c0 2024-04-27T03:55:18.985 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:17.903+0000 7f91256d5700 1 -- 172.21.15.46:0/522454049 --> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9120062330 con 0x7f9108038780 2024-04-27T03:55:18.985 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:18.149+0000 7f9107fff700 1 -- 172.21.15.46:0/522454049 <== mgr.14162 v2:172.21.15.46:6800/3280787672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f9120062330 con 0x7f9108038780 2024-04-27T03:55:18.985 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:18.151+0000 7f91256d5700 1 -- 172.21.15.46:0/522454049 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f9108038780 msgr2=0x7f910803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:18.985 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:18.151+0000 7f91256d5700 1 --2- 172.21.15.46:0/522454049 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f9108038780 0x7f910803ac40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9114006fd0 tx=0x7f9114006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:18.985 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:18.151+0000 7f91256d5700 1 -- 172.21.15.46:0/522454049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91201065c0 msgr2=0x7f912019a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:18.985 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:18.151+0000 7f91256d5700 1 --2- 172.21.15.46:0/522454049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91201065c0 0x7f912019a350 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9110009fb0 tx=0x7f9110011890 comp rx=0 tx=0).stop 2024-04-27T03:55:18.985 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:18.151+0000 7f91256d5700 1 -- 172.21.15.46:0/522454049 shutdown_connections 2024-04-27T03:55:18.985 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:18.151+0000 7f91256d5700 1 --2- 172.21.15.46:0/522454049 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f9108038780 0x7f910803ac40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:18.985 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:18.151+0000 7f91256d5700 1 --2- 172.21.15.46:0/522454049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91201065c0 0x7f912019a350 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:18.985 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:18.151+0000 7f91256d5700 1 -- 172.21.15.46:0/522454049 >> 172.21.15.46:0/522454049 conn(0x7f9120101be0 msgr2=0x7f9120102730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:18.985 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:18.152+0000 7f91256d5700 1 -- 172.21.15.46:0/522454049 shutdown_connections 2024-04-27T03:55:18.986 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:18.152+0000 7f91256d5700 1 -- 172.21.15.46:0/522454049 wait complete. 2024-04-27T03:55:18.986 INFO:teuthology.orchestra.run.smithi046.stdout:Fetching dashboard port number... 2024-04-27T03:55:20.388 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:20 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/558036595' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-27T03:55:20.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 8443 2024-04-27T03:55:20.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.826+0000 7f2a10f3c700 1 Processor -- start 2024-04-27T03:55:20.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.826+0000 7f2a10f3c700 1 -- start start 2024-04-27T03:55:20.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.826+0000 7f2a10f3c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0c1065f0 0x7f2a0c106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:20.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.826+0000 7f2a10f3c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c106fe0 con 0x7f2a0c1065f0 2024-04-27T03:55:20.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.827+0000 7f2a0a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0c1065f0 0x7f2a0c106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:20.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.827+0000 7f2a0a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0c1065f0 0x7f2a0c106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37582/0 (socket says 172.21.15.46:37582) 2024-04-27T03:55:20.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.827+0000 7f2a0a59c700 1 -- 172.21.15.46:0/3588724763 learned_addr learned my addr 172.21.15.46:0/3588724763 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:20.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.827+0000 7f2a0a59c700 1 -- 172.21.15.46:0/3588724763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c1077a0 con 0x7f2a0c1065f0 2024-04-27T03:55:20.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.827+0000 7f2a0a59c700 1 --2- 172.21.15.46:0/3588724763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0c1065f0 0x7f2a0c106a10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2a04009e80 tx=0x7f2a0400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f903311d9d125e3c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:20.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.827+0000 7f2a0959a700 1 -- 172.21.15.46:0/3588724763 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a0400e070 con 0x7f2a0c1065f0 2024-04-27T03:55:20.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.827+0000 7f2a0959a700 1 -- 172.21.15.46:0/3588724763 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a0400b8c0 con 0x7f2a0c1065f0 2024-04-27T03:55:20.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.827+0000 7f2a0959a700 1 -- 172.21.15.46:0/3588724763 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a040044d0 con 0x7f2a0c1065f0 2024-04-27T03:55:20.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.828+0000 7f2a10f3c700 1 -- 172.21.15.46:0/3588724763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0c1065f0 msgr2=0x7f2a0c106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:20.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.828+0000 7f2a10f3c700 1 --2- 172.21.15.46:0/3588724763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0c1065f0 0x7f2a0c106a10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2a04009e80 tx=0x7f2a0400b330 comp rx=0 tx=0).stop 2024-04-27T03:55:20.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.828+0000 7f2a10f3c700 1 -- 172.21.15.46:0/3588724763 shutdown_connections 2024-04-27T03:55:20.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.828+0000 7f2a10f3c700 1 --2- 172.21.15.46:0/3588724763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0c1065f0 0x7f2a0c106a10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:20.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.828+0000 7f2a10f3c700 1 -- 172.21.15.46:0/3588724763 >> 172.21.15.46:0/3588724763 conn(0x7f2a0c101bd0 msgr2=0x7f2a0c103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:20.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.828+0000 7f2a10f3c700 1 -- 172.21.15.46:0/3588724763 shutdown_connections 2024-04-27T03:55:20.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.828+0000 7f2a10f3c700 1 -- 172.21.15.46:0/3588724763 wait complete. 2024-04-27T03:55:20.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.829+0000 7f2a10f3c700 1 Processor -- start 2024-04-27T03:55:20.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.829+0000 7f2a10f3c700 1 -- start start 2024-04-27T03:55:20.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.829+0000 7f2a10f3c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0c1065f0 0x7f2a0c07bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:20.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.829+0000 7f2a10f3c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c07c080 con 0x7f2a0c1065f0 2024-04-27T03:55:20.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.829+0000 7f2a0a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0c1065f0 0x7f2a0c07bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:20.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.829+0000 7f2a0a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0c1065f0 0x7f2a0c07bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37598/0 (socket says 172.21.15.46:37598) 2024-04-27T03:55:20.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.829+0000 7f2a0a59c700 1 -- 172.21.15.46:0/558036595 learned_addr learned my addr 172.21.15.46:0/558036595 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:20.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.829+0000 7f2a0a59c700 1 -- 172.21.15.46:0/558036595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a04009b30 con 0x7f2a0c1065f0 2024-04-27T03:55:20.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.829+0000 7f2a0a59c700 1 --2- 172.21.15.46:0/558036595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0c1065f0 0x7f2a0c07bb40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2a04000f80 tx=0x7f2a0400bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:20.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.830+0000 7f2a037fe700 1 -- 172.21.15.46:0/558036595 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a0400e040 con 0x7f2a0c1065f0 2024-04-27T03:55:20.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.830+0000 7f2a037fe700 1 -- 172.21.15.46:0/558036595 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a04003c30 con 0x7f2a0c1065f0 2024-04-27T03:55:20.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.830+0000 7f2a10f3c700 1 -- 172.21.15.46:0/558036595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0c07a2c0 con 0x7f2a0c1065f0 2024-04-27T03:55:20.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.830+0000 7f2a037fe700 1 -- 172.21.15.46:0/558036595 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a04012670 con 0x7f2a0c1065f0 2024-04-27T03:55:20.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.830+0000 7f2a10f3c700 1 -- 172.21.15.46:0/558036595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0c07a7c0 con 0x7f2a0c1065f0 2024-04-27T03:55:20.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.830+0000 7f2a037fe700 1 -- 172.21.15.46:0/558036595 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) v1 ==== 45569+0+0 (secure 0 0 0) 0x7f2a04019040 con 0x7f2a0c1065f0 2024-04-27T03:55:20.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.831+0000 7f2a10f3c700 1 -- 172.21.15.46:0/558036595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0c062510 con 0x7f2a0c1065f0 2024-04-27T03:55:20.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.831+0000 7f2a037fe700 1 --2- 172.21.15.46:0/558036595 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f29f8038770 0x7f29f803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:20.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.831+0000 7f2a037fe700 1 -- 172.21.15.46:0/558036595 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2a0404c4d0 con 0x7f2a0c1065f0 2024-04-27T03:55:20.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.831+0000 7f2a09d9b700 1 --2- 172.21.15.46:0/558036595 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f29f8038770 0x7f29f803ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:20.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.831+0000 7f2a09d9b700 1 --2- 172.21.15.46:0/558036595 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f29f8038770 0x7f29f803ac30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f29f4006fd0 tx=0x7f29f4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:20.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:19.836+0000 7f2a037fe700 1 -- 172.21.15.46:0/558036595 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f2a04011570 con 0x7f2a0c1065f0 2024-04-27T03:55:20.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:20.037+0000 7f2a10f3c700 1 -- 172.21.15.46:0/558036595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f2a0c07ac50 con 0x7f2a0c1065f0 2024-04-27T03:55:20.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:20.038+0000 7f2a037fe700 1 -- 172.21.15.46:0/558036595 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f2a0401f070 con 0x7f2a0c1065f0 2024-04-27T03:55:20.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:20.040+0000 7f2a10f3c700 1 -- 172.21.15.46:0/558036595 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f29f8038770 msgr2=0x7f29f803ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:20.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:20.040+0000 7f2a10f3c700 1 --2- 172.21.15.46:0/558036595 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f29f8038770 0x7f29f803ac30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f29f4006fd0 tx=0x7f29f4006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:20.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:20.040+0000 7f2a10f3c700 1 -- 172.21.15.46:0/558036595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0c1065f0 msgr2=0x7f2a0c07bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:20.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:20.040+0000 7f2a10f3c700 1 --2- 172.21.15.46:0/558036595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0c1065f0 0x7f2a0c07bb40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2a04000f80 tx=0x7f2a0400bd10 comp rx=0 tx=0).stop 2024-04-27T03:55:20.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:20.040+0000 7f2a10f3c700 1 -- 172.21.15.46:0/558036595 shutdown_connections 2024-04-27T03:55:20.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:20.040+0000 7f2a10f3c700 1 --2- 172.21.15.46:0/558036595 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f29f8038770 0x7f29f803ac30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:20.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:20.040+0000 7f2a10f3c700 1 --2- 172.21.15.46:0/558036595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0c1065f0 0x7f2a0c07bb40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:20.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:20.040+0000 7f2a10f3c700 1 -- 172.21.15.46:0/558036595 >> 172.21.15.46:0/558036595 conn(0x7f2a0c101bd0 msgr2=0x7f2a0c102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:20.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:20.040+0000 7f2a10f3c700 1 -- 172.21.15.46:0/558036595 shutdown_connections 2024-04-27T03:55:20.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:20.040+0000 7f2a10f3c700 1 -- 172.21.15.46:0/558036595 wait complete. 2024-04-27T03:55:20.661 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-27T03:55:20.661 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2024-04-27T03:55:20.668 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-27T03:55:20.669 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-04-27T03:55:20.669 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2024-04-27T03:55:20.669 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-27T03:55:20.670 INFO:teuthology.orchestra.run.smithi046.stdout:Ceph Dashboard is now available at: 2024-04-27T03:55:20.670 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:55:20.670 INFO:teuthology.orchestra.run.smithi046.stdout: URL: https://smithi046:8443/ 2024-04-27T03:55:20.670 INFO:teuthology.orchestra.run.smithi046.stdout: User: admin 2024-04-27T03:55:20.670 INFO:teuthology.orchestra.run.smithi046.stdout: Password: xuzgehzj1a 2024-04-27T03:55:20.671 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:55:20.671 INFO:teuthology.orchestra.run.smithi046.stdout:Saving cluster configuration to /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/config directory 2024-04-27T03:55:20.673 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling autotune for osd_memory_target 2024-04-27T03:55:21.281 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:21 smithi046 ceph-mon[26899]: mgrmap e13: smithi046.qrfdvr(active, since 6s) 2024-04-27T03:55:21.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.067+0000 7f956a729700 1 Processor -- start 2024-04-27T03:55:21.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.067+0000 7f956a729700 1 -- start start 2024-04-27T03:55:21.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.068+0000 7f956a729700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95641065e0 0x7f9564106a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:21.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.068+0000 7f956a729700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9564106fd0 con 0x7f95641065e0 2024-04-27T03:55:21.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.068+0000 7f9563fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95641065e0 0x7f9564106a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:21.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.068+0000 7f9563fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95641065e0 0x7f9564106a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46998/0 (socket says 172.21.15.46:46998) 2024-04-27T03:55:21.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.068+0000 7f9563fff700 1 -- 172.21.15.46:0/578054240 learned_addr learned my addr 172.21.15.46:0/578054240 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:21.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.068+0000 7f9563fff700 1 -- 172.21.15.46:0/578054240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9564107790 con 0x7f95641065e0 2024-04-27T03:55:21.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.068+0000 7f9563fff700 1 --2- 172.21.15.46:0/578054240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95641065e0 0x7f9564106a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f95540098a0 tx=0x7f9554009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adfbde502d93f2c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:21.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.068+0000 7f9562ffd700 1 -- 172.21.15.46:0/578054240 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9554004180 con 0x7f95641065e0 2024-04-27T03:55:21.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.068+0000 7f9562ffd700 1 -- 172.21.15.46:0/578054240 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95540042e0 con 0x7f95641065e0 2024-04-27T03:55:21.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.068+0000 7f9562ffd700 1 -- 172.21.15.46:0/578054240 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9554011b10 con 0x7f95641065e0 2024-04-27T03:55:21.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.069+0000 7f956a729700 1 -- 172.21.15.46:0/578054240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95641065e0 msgr2=0x7f9564106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:21.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.069+0000 7f956a729700 1 --2- 172.21.15.46:0/578054240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95641065e0 0x7f9564106a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f95540098a0 tx=0x7f9554009bb0 comp rx=0 tx=0).stop 2024-04-27T03:55:21.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.069+0000 7f956a729700 1 -- 172.21.15.46:0/578054240 shutdown_connections 2024-04-27T03:55:21.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.069+0000 7f956a729700 1 --2- 172.21.15.46:0/578054240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95641065e0 0x7f9564106a00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:21.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.069+0000 7f956a729700 1 -- 172.21.15.46:0/578054240 >> 172.21.15.46:0/578054240 conn(0x7f9564101b80 msgr2=0x7f9564103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:21.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.069+0000 7f956a729700 1 -- 172.21.15.46:0/578054240 shutdown_connections 2024-04-27T03:55:21.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.069+0000 7f956a729700 1 -- 172.21.15.46:0/578054240 wait complete. 2024-04-27T03:55:21.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.070+0000 7f956a729700 1 Processor -- start 2024-04-27T03:55:21.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.070+0000 7f956a729700 1 -- start start 2024-04-27T03:55:21.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.070+0000 7f956a729700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95641065e0 0x7f956419a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:21.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.070+0000 7f956a729700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956419a8c0 con 0x7f95641065e0 2024-04-27T03:55:21.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.070+0000 7f9563fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95641065e0 0x7f956419a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:21.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.070+0000 7f9563fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95641065e0 0x7f956419a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47002/0 (socket says 172.21.15.46:47002) 2024-04-27T03:55:21.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.070+0000 7f9563fff700 1 -- 172.21.15.46:0/1562880900 learned_addr learned my addr 172.21.15.46:0/1562880900 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:21.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.070+0000 7f9563fff700 1 -- 172.21.15.46:0/1562880900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9554009550 con 0x7f95641065e0 2024-04-27T03:55:21.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.070+0000 7f9563fff700 1 --2- 172.21.15.46:0/1562880900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95641065e0 0x7f956419a380 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9554000c00 tx=0x7f9554011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:21.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.071+0000 7f95617fa700 1 -- 172.21.15.46:0/1562880900 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95540038b0 con 0x7f95641065e0 2024-04-27T03:55:21.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.071+0000 7f95617fa700 1 -- 172.21.15.46:0/1562880900 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9554003a10 con 0x7f95641065e0 2024-04-27T03:55:21.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.071+0000 7f95617fa700 1 -- 172.21.15.46:0/1562880900 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9554024780 con 0x7f95641065e0 2024-04-27T03:55:21.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.071+0000 7f956a729700 1 -- 172.21.15.46:0/1562880900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956419aac0 con 0x7f95641065e0 2024-04-27T03:55:21.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.071+0000 7f956a729700 1 -- 172.21.15.46:0/1562880900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956419af60 con 0x7f95641065e0 2024-04-27T03:55:21.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.072+0000 7f95617fa700 1 -- 172.21.15.46:0/1562880900 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f955402d3f0 con 0x7f95641065e0 2024-04-27T03:55:21.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.072+0000 7f956a729700 1 -- 172.21.15.46:0/1562880900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956410a9a0 con 0x7f95641065e0 2024-04-27T03:55:21.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.072+0000 7f95617fa700 1 --2- 172.21.15.46:0/1562880900 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f954c038810 0x7f954c03acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:21.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.072+0000 7f95617fa700 1 -- 172.21.15.46:0/1562880900 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f955404c900 con 0x7f95641065e0 2024-04-27T03:55:21.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.072+0000 7f95637fe700 1 --2- 172.21.15.46:0/1562880900 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f954c038810 0x7f954c03acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:21.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.072+0000 7f95637fe700 1 --2- 172.21.15.46:0/1562880900 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f954c038810 0x7f954c03acd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9558006fd0 tx=0x7f9558006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:21.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.077+0000 7f95617fa700 1 -- 172.21.15.46:0/1562880900 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f9554010bd0 con 0x7f95641065e0 2024-04-27T03:55:21.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.279+0000 7f956a729700 1 -- 172.21.15.46:0/1562880900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f956419d850 con 0x7f95641065e0 2024-04-27T03:55:21.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.279+0000 7f95617fa700 1 -- 172.21.15.46:0/1562880900 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f95540107f0 con 0x7f95641065e0 2024-04-27T03:55:21.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.282+0000 7f956a729700 1 -- 172.21.15.46:0/1562880900 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f954c038810 msgr2=0x7f954c03acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:21.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.282+0000 7f956a729700 1 --2- 172.21.15.46:0/1562880900 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f954c038810 0x7f954c03acd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9558006fd0 tx=0x7f9558006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:21.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.282+0000 7f956a729700 1 -- 172.21.15.46:0/1562880900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95641065e0 msgr2=0x7f956419a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:21.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.282+0000 7f956a729700 1 --2- 172.21.15.46:0/1562880900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95641065e0 0x7f956419a380 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9554000c00 tx=0x7f9554011890 comp rx=0 tx=0).stop 2024-04-27T03:55:21.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.282+0000 7f956a729700 1 -- 172.21.15.46:0/1562880900 shutdown_connections 2024-04-27T03:55:21.916 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.282+0000 7f956a729700 1 --2- 172.21.15.46:0/1562880900 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f954c038810 0x7f954c03acd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:21.916 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.282+0000 7f956a729700 1 --2- 172.21.15.46:0/1562880900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95641065e0 0x7f956419a380 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:21.916 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.282+0000 7f956a729700 1 -- 172.21.15.46:0/1562880900 >> 172.21.15.46:0/1562880900 conn(0x7f9564101b80 msgr2=0x7f9564102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:21.916 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.282+0000 7f956a729700 1 -- 172.21.15.46:0/1562880900 shutdown_connections 2024-04-27T03:55:21.916 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:21.282+0000 7f956a729700 1 -- 172.21.15.46:0/1562880900 wait complete. 2024-04-27T03:55:23.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.335+0000 7effd8fec700 1 Processor -- start 2024-04-27T03:55:23.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.335+0000 7effd8fec700 1 -- start start 2024-04-27T03:55:23.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.335+0000 7effd8fec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd41066a0 0x7effd4106ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:23.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.335+0000 7effd8fec700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd4107090 con 0x7effd41066a0 2024-04-27T03:55:23.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.336+0000 7effd259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd41066a0 0x7effd4106ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:23.216 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.336+0000 7effd259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd41066a0 0x7effd4106ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47012/0 (socket says 172.21.15.46:47012) 2024-04-27T03:55:23.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.336+0000 7effd259c700 1 -- 172.21.15.46:0/300816660 learned_addr learned my addr 172.21.15.46:0/300816660 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:23.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.336+0000 7effd259c700 1 -- 172.21.15.46:0/300816660 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd4107850 con 0x7effd41066a0 2024-04-27T03:55:23.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.336+0000 7effd259c700 1 --2- 172.21.15.46:0/300816660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd41066a0 0x7effd4106ac0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7effc80098a0 tx=0x7effc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b45df76039854d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:23.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.336+0000 7effd159a700 1 -- 172.21.15.46:0/300816660 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effc8004180 con 0x7effd41066a0 2024-04-27T03:55:23.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.336+0000 7effd159a700 1 -- 172.21.15.46:0/300816660 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effc80042e0 con 0x7effd41066a0 2024-04-27T03:55:23.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.336+0000 7effd159a700 1 -- 172.21.15.46:0/300816660 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effc8011b10 con 0x7effd41066a0 2024-04-27T03:55:23.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.337+0000 7effd8fec700 1 -- 172.21.15.46:0/300816660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd41066a0 msgr2=0x7effd4106ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:23.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.337+0000 7effd8fec700 1 --2- 172.21.15.46:0/300816660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd41066a0 0x7effd4106ac0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7effc80098a0 tx=0x7effc8009bb0 comp rx=0 tx=0).stop 2024-04-27T03:55:23.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.337+0000 7effd8fec700 1 -- 172.21.15.46:0/300816660 shutdown_connections 2024-04-27T03:55:23.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.337+0000 7effd8fec700 1 --2- 172.21.15.46:0/300816660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd41066a0 0x7effd4106ac0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:23.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.337+0000 7effd8fec700 1 -- 172.21.15.46:0/300816660 >> 172.21.15.46:0/300816660 conn(0x7effd4101c40 msgr2=0x7effd4104080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:23.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.337+0000 7effd8fec700 1 -- 172.21.15.46:0/300816660 shutdown_connections 2024-04-27T03:55:23.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.337+0000 7effd8fec700 1 -- 172.21.15.46:0/300816660 wait complete. 2024-04-27T03:55:23.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.338+0000 7effd8fec700 1 Processor -- start 2024-04-27T03:55:23.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.338+0000 7effd8fec700 1 -- start start 2024-04-27T03:55:23.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.338+0000 7effd8fec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd41066a0 0x7effd407bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:23.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.338+0000 7effd8fec700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd407c080 con 0x7effd41066a0 2024-04-27T03:55:23.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.338+0000 7effd259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd41066a0 0x7effd407bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:23.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.338+0000 7effd259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd41066a0 0x7effd407bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47014/0 (socket says 172.21.15.46:47014) 2024-04-27T03:55:23.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.338+0000 7effd259c700 1 -- 172.21.15.46:0/1599618784 learned_addr learned my addr 172.21.15.46:0/1599618784 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:23.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.338+0000 7effd259c700 1 -- 172.21.15.46:0/1599618784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc8009550 con 0x7effd41066a0 2024-04-27T03:55:23.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.339+0000 7effd259c700 1 --2- 172.21.15.46:0/1599618784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd41066a0 0x7effd407bb40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7effc8009fb0 tx=0x7effc8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:23.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.339+0000 7effc37fe700 1 -- 172.21.15.46:0/1599618784 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effc80038b0 con 0x7effd41066a0 2024-04-27T03:55:23.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.339+0000 7effd8fec700 1 -- 172.21.15.46:0/1599618784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd407a2c0 con 0x7effd41066a0 2024-04-27T03:55:23.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.339+0000 7effc37fe700 1 -- 172.21.15.46:0/1599618784 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effc8003a10 con 0x7effd41066a0 2024-04-27T03:55:23.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.339+0000 7effc37fe700 1 -- 172.21.15.46:0/1599618784 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effc8019770 con 0x7effd41066a0 2024-04-27T03:55:23.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.339+0000 7effd8fec700 1 -- 172.21.15.46:0/1599618784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd407a760 con 0x7effd41066a0 2024-04-27T03:55:23.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.340+0000 7effc37fe700 1 -- 172.21.15.46:0/1599618784 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7effc80198d0 con 0x7effd41066a0 2024-04-27T03:55:23.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.340+0000 7effd8fec700 1 -- 172.21.15.46:0/1599618784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd4062510 con 0x7effd41066a0 2024-04-27T03:55:23.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.340+0000 7effc37fe700 1 --2- 172.21.15.46:0/1599618784 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7effbc0387d0 0x7effbc03ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:23.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.340+0000 7effc37fe700 1 -- 172.21.15.46:0/1599618784 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7effc804cc00 con 0x7effd41066a0 2024-04-27T03:55:23.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.340+0000 7effd1d9b700 1 --2- 172.21.15.46:0/1599618784 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7effbc0387d0 0x7effbc03ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:23.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.340+0000 7effd1d9b700 1 --2- 172.21.15.46:0/1599618784 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7effbc0387d0 0x7effbc03ac90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7effc4006fd0 tx=0x7effc4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:23.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.345+0000 7effc37fe700 1 -- 172.21.15.46:0/1599618784 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7effc8053080 con 0x7effd41066a0 2024-04-27T03:55:23.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.656+0000 7effd8fec700 1 -- 172.21.15.46:0/1599618784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7effd407ac50 con 0x7effd41066a0 2024-04-27T03:55:23.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.659+0000 7effc37fe700 1 -- 172.21.15.46:0/1599618784 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7effc8019e00 con 0x7effd41066a0 2024-04-27T03:55:23.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.661+0000 7effd8fec700 1 -- 172.21.15.46:0/1599618784 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7effbc0387d0 msgr2=0x7effbc03ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:23.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.661+0000 7effd8fec700 1 --2- 172.21.15.46:0/1599618784 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7effbc0387d0 0x7effbc03ac90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7effc4006fd0 tx=0x7effc4006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:23.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.661+0000 7effd8fec700 1 -- 172.21.15.46:0/1599618784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd41066a0 msgr2=0x7effd407bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:23.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.661+0000 7effd8fec700 1 --2- 172.21.15.46:0/1599618784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd41066a0 0x7effd407bb40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7effc8009fb0 tx=0x7effc8011890 comp rx=0 tx=0).stop 2024-04-27T03:55:23.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.661+0000 7effd8fec700 1 -- 172.21.15.46:0/1599618784 shutdown_connections 2024-04-27T03:55:23.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.661+0000 7effd8fec700 1 --2- 172.21.15.46:0/1599618784 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7effbc0387d0 0x7effbc03ac90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:23.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.661+0000 7effd8fec700 1 --2- 172.21.15.46:0/1599618784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd41066a0 0x7effd407bb40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:23.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.661+0000 7effd8fec700 1 -- 172.21.15.46:0/1599618784 >> 172.21.15.46:0/1599618784 conn(0x7effd4101c40 msgr2=0x7effd41027a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:23.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.661+0000 7effd8fec700 1 -- 172.21.15.46:0/1599618784 shutdown_connections 2024-04-27T03:55:23.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-27T03:55:22.661+0000 7effd8fec700 1 -- 172.21.15.46:0/1599618784 wait complete. 2024-04-27T03:55:23.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-27T03:55:23.220 INFO:teuthology.orchestra.run.smithi046.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-27T03:55:23.221 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:55:23.221 INFO:teuthology.orchestra.run.smithi046.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-27T03:55:23.221 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:55:23.221 INFO:teuthology.orchestra.run.smithi046.stdout:Or, if you are only running a single cluster on this host: 2024-04-27T03:55:23.221 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:55:23.221 INFO:teuthology.orchestra.run.smithi046.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-27T03:55:23.221 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:55:23.221 INFO:teuthology.orchestra.run.smithi046.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-27T03:55:23.221 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:55:23.221 INFO:teuthology.orchestra.run.smithi046.stdout: ceph telemetry on 2024-04-27T03:55:23.221 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:55:23.221 INFO:teuthology.orchestra.run.smithi046.stdout:For more information see: 2024-04-27T03:55:23.221 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:55:23.221 INFO:teuthology.orchestra.run.smithi046.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-27T03:55:23.222 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:55:23.222 INFO:teuthology.orchestra.run.smithi046.stdout:Bootstrap complete. 2024-04-27T03:55:23.247 INFO:tasks.cephadm:Fetching config... 2024-04-27T03:55:23.247 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T03:55:23.247 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-27T03:55:23.305 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-27T03:55:23.305 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T03:55:23.305 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-27T03:55:23.361 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-27T03:55:23.361 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T03:55:23.361 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/keyring of=/dev/stdout 2024-04-27T03:55:23.427 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-27T03:55:23.427 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T03:55:23.427 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-27T03:55:23.483 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-27T03:55:23.483 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCweCM9Acgf13VvpFVvlKocoNJjU3vbp+GUML+rGDel6vYoVSAquz08wg/kdr7Mqr+5MQ2k7tdZ3515qvjm+JXC6DAq8gj0TV9igHR+sNSCCGqNSr14AOqiEbmCkGIeP0bOUnoUqipcvqXXozZ1Ed7hFxVxcwODpSX9qaqhuyV+ZsQhGT8mboHSqpcgnS2z/HBIh7Srd3UaY0d1p/bAETk8K+aymS/zS6wtbNA5YM9jfBIaQ1tnYcPii30jC7BFg6UxRtg5j+sfTAf3b67jBm7n51YRLA1ZL/y4cu2qpzYRoN6u0dCSmOzdbBiFylTKSj0IOvrhmGSqNZ30hfGMuHEAGQuQ13S79eUaP9n0ssqFDcUXp3z3rqvFXNzymxP41i94Wc61bc0sYvqC863l6ixgBXTR3cDXD6vMz0y/dlwhHvo5z9yqEerxwRdt0+fmgkpZITSAVU3cEBYIOd3JJOBuzEU/MKKsa/n1g7ZLyYKsWDqhRUCUHp8WAZ7PLiO6Dz8= ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-27T03:55:23.562 INFO:teuthology.orchestra.run.smithi046.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCweCM9Acgf13VvpFVvlKocoNJjU3vbp+GUML+rGDel6vYoVSAquz08wg/kdr7Mqr+5MQ2k7tdZ3515qvjm+JXC6DAq8gj0TV9igHR+sNSCCGqNSr14AOqiEbmCkGIeP0bOUnoUqipcvqXXozZ1Ed7hFxVxcwODpSX9qaqhuyV+ZsQhGT8mboHSqpcgnS2z/HBIh7Srd3UaY0d1p/bAETk8K+aymS/zS6wtbNA5YM9jfBIaQ1tnYcPii30jC7BFg6UxRtg5j+sfTAf3b67jBm7n51YRLA1ZL/y4cu2qpzYRoN6u0dCSmOzdbBiFylTKSj0IOvrhmGSqNZ30hfGMuHEAGQuQ13S79eUaP9n0ssqFDcUXp3z3rqvFXNzymxP41i94Wc61bc0sYvqC863l6ixgBXTR3cDXD6vMz0y/dlwhHvo5z9yqEerxwRdt0+fmgkpZITSAVU3cEBYIOd3JJOBuzEU/MKKsa/n1g7ZLyYKsWDqhRUCUHp8WAZ7PLiO6Dz8= ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968 2024-04-27T03:55:23.574 DEBUG:teuthology.orchestra.run.smithi073:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCweCM9Acgf13VvpFVvlKocoNJjU3vbp+GUML+rGDel6vYoVSAquz08wg/kdr7Mqr+5MQ2k7tdZ3515qvjm+JXC6DAq8gj0TV9igHR+sNSCCGqNSr14AOqiEbmCkGIeP0bOUnoUqipcvqXXozZ1Ed7hFxVxcwODpSX9qaqhuyV+ZsQhGT8mboHSqpcgnS2z/HBIh7Srd3UaY0d1p/bAETk8K+aymS/zS6wtbNA5YM9jfBIaQ1tnYcPii30jC7BFg6UxRtg5j+sfTAf3b67jBm7n51YRLA1ZL/y4cu2qpzYRoN6u0dCSmOzdbBiFylTKSj0IOvrhmGSqNZ30hfGMuHEAGQuQ13S79eUaP9n0ssqFDcUXp3z3rqvFXNzymxP41i94Wc61bc0sYvqC863l6ixgBXTR3cDXD6vMz0y/dlwhHvo5z9yqEerxwRdt0+fmgkpZITSAVU3cEBYIOd3JJOBuzEU/MKKsa/n1g7ZLyYKsWDqhRUCUHp8WAZ7PLiO6Dz8= ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-27T03:55:23.614 INFO:teuthology.orchestra.run.smithi073.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCweCM9Acgf13VvpFVvlKocoNJjU3vbp+GUML+rGDel6vYoVSAquz08wg/kdr7Mqr+5MQ2k7tdZ3515qvjm+JXC6DAq8gj0TV9igHR+sNSCCGqNSr14AOqiEbmCkGIeP0bOUnoUqipcvqXXozZ1Ed7hFxVxcwODpSX9qaqhuyV+ZsQhGT8mboHSqpcgnS2z/HBIh7Srd3UaY0d1p/bAETk8K+aymS/zS6wtbNA5YM9jfBIaQ1tnYcPii30jC7BFg6UxRtg5j+sfTAf3b67jBm7n51YRLA1ZL/y4cu2qpzYRoN6u0dCSmOzdbBiFylTKSj0IOvrhmGSqNZ30hfGMuHEAGQuQ13S79eUaP9n0ssqFDcUXp3z3rqvFXNzymxP41i94Wc61bc0sYvqC863l6ixgBXTR3cDXD6vMz0y/dlwhHvo5z9yqEerxwRdt0+fmgkpZITSAVU3cEBYIOd3JJOBuzEU/MKKsa/n1g7ZLyYKsWDqhRUCUHp8WAZ7PLiO6Dz8= ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968 2024-04-27T03:55:23.625 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-27T03:55:23.837 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T03:55:23.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:23 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/1599618784' entity='client.admin' 2024-04-27T03:55:26.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.120+0000 7f91a2eed700 1 -- 172.21.15.46:0/3352694574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f919c100870 msgr2=0x7f919c100c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:26.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.120+0000 7f91a2eed700 1 --2- 172.21.15.46:0/3352694574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f919c100870 0x7f919c100c90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9190009910 tx=0x7f9190009c20 comp rx=0 tx=0).stop 2024-04-27T03:55:26.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.120+0000 7f91a2eed700 1 -- 172.21.15.46:0/3352694574 shutdown_connections 2024-04-27T03:55:26.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.120+0000 7f91a2eed700 1 --2- 172.21.15.46:0/3352694574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f919c100870 0x7f919c100c90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:26.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.120+0000 7f91a2eed700 1 -- 172.21.15.46:0/3352694574 >> 172.21.15.46:0/3352694574 conn(0x7f919c0fbe30 msgr2=0x7f919c0fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:26.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.120+0000 7f91a2eed700 1 -- 172.21.15.46:0/3352694574 shutdown_connections 2024-04-27T03:55:26.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.121+0000 7f91a2eed700 1 -- 172.21.15.46:0/3352694574 wait complete. 2024-04-27T03:55:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.121+0000 7f91a2eed700 1 Processor -- start 2024-04-27T03:55:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.121+0000 7f91a2eed700 1 -- start start 2024-04-27T03:55:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.121+0000 7f91a2eed700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f919c100870 0x7f919c074b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.121+0000 7f91a2eed700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919c075090 con 0x7f919c100870 2024-04-27T03:55:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.121+0000 7f91a0c89700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f919c100870 0x7f919c074b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.121+0000 7f91a0c89700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f919c100870 0x7f919c074b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47034/0 (socket says 172.21.15.46:47034) 2024-04-27T03:55:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.121+0000 7f91a0c89700 1 -- 172.21.15.46:0/1027458912 learned_addr learned my addr 172.21.15.46:0/1027458912 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.122+0000 7f91a0c89700 1 -- 172.21.15.46:0/1027458912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91900095f0 con 0x7f919c100870 2024-04-27T03:55:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.122+0000 7f91a0c89700 1 --2- 172.21.15.46:0/1027458912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f919c100870 0x7f919c074b50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9190006010 tx=0x7f91900050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.122+0000 7f9199ffb700 1 -- 172.21.15.46:0/1027458912 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f919001d070 con 0x7f919c100870 2024-04-27T03:55:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.122+0000 7f91a2eed700 1 -- 172.21.15.46:0/1027458912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919c0732d0 con 0x7f919c100870 2024-04-27T03:55:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.122+0000 7f9199ffb700 1 -- 172.21.15.46:0/1027458912 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f919000be20 con 0x7f919c100870 2024-04-27T03:55:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.122+0000 7f9199ffb700 1 -- 172.21.15.46:0/1027458912 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9190022de0 con 0x7f919c100870 2024-04-27T03:55:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.122+0000 7f91a2eed700 1 -- 172.21.15.46:0/1027458912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919c073770 con 0x7f919c100870 2024-04-27T03:55:26.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.123+0000 7f9199ffb700 1 -- 172.21.15.46:0/1027458912 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f919000f4b0 con 0x7f919c100870 2024-04-27T03:55:26.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.123+0000 7f9199ffb700 1 --2- 172.21.15.46:0/1027458912 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f918c0387b0 0x7f918c03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:26.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.123+0000 7f9199ffb700 1 -- 172.21.15.46:0/1027458912 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f919004c880 con 0x7f919c100870 2024-04-27T03:55:26.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.123+0000 7f91a2eed700 1 -- 172.21.15.46:0/1027458912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919c106fd0 con 0x7f919c100870 2024-04-27T03:55:26.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.123+0000 7f919bfff700 1 --2- 172.21.15.46:0/1027458912 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f918c0387b0 0x7f918c03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:26.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.124+0000 7f919bfff700 1 --2- 172.21.15.46:0/1027458912 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f918c0387b0 0x7f918c03ac70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9188006fd0 tx=0x7f9188006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:26.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.129+0000 7f9199ffb700 1 -- 172.21.15.46:0/1027458912 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f9190027020 con 0x7f919c100870 2024-04-27T03:55:26.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.337+0000 7f91a2eed700 1 -- 172.21.15.46:0/1027458912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f919c18f080 con 0x7f919c100870 2024-04-27T03:55:26.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.341+0000 7f9199ffb700 1 -- 172.21.15.46:0/1027458912 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9190017440 con 0x7f919c100870 2024-04-27T03:55:26.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.343+0000 7f91a2eed700 1 -- 172.21.15.46:0/1027458912 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f918c0387b0 msgr2=0x7f918c03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:26.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.343+0000 7f91a2eed700 1 --2- 172.21.15.46:0/1027458912 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f918c0387b0 0x7f918c03ac70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9188006fd0 tx=0x7f9188006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:26.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.343+0000 7f91a2eed700 1 -- 172.21.15.46:0/1027458912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f919c100870 msgr2=0x7f919c074b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:26.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.343+0000 7f91a2eed700 1 --2- 172.21.15.46:0/1027458912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f919c100870 0x7f919c074b50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9190006010 tx=0x7f91900050b0 comp rx=0 tx=0).stop 2024-04-27T03:55:26.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.343+0000 7f91a2eed700 1 -- 172.21.15.46:0/1027458912 shutdown_connections 2024-04-27T03:55:26.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.343+0000 7f91a2eed700 1 --2- 172.21.15.46:0/1027458912 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f918c0387b0 0x7f918c03ac70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:26.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.343+0000 7f91a2eed700 1 --2- 172.21.15.46:0/1027458912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f919c100870 0x7f919c074b50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:26.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.343+0000 7f91a2eed700 1 -- 172.21.15.46:0/1027458912 >> 172.21.15.46:0/1027458912 conn(0x7f919c0fbe30 msgr2=0x7f919c0fc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:26.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.343+0000 7f91a2eed700 1 -- 172.21.15.46:0/1027458912 shutdown_connections 2024-04-27T03:55:26.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:26.343+0000 7f91a2eed700 1 -- 172.21.15.46:0/1027458912 wait complete. 2024-04-27T03:55:26.465 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:26 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:26.465 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:26 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:26.466 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:26 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi046", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:55:26.466 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:26 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:26.466 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:26 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi046", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-27T03:55:26.466 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:26 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi046", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-27T03:55:26.466 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:26 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:55:26.466 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:26 smithi046 ceph-mon[26899]: Deploying daemon ceph-exporter.smithi046 on smithi046 2024-04-27T03:55:27.232 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-27T03:55:27.232 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-27T03:55:27.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:27 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/1027458912' entity='client.admin' 2024-04-27T03:55:27.874 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T03:55:30.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.115+0000 7f9cab0a9700 1 -- 172.21.15.46:0/1777890851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca40ff760 msgr2=0x7f9ca40ffb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:30.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.115+0000 7f9cab0a9700 1 --2- 172.21.15.46:0/1777890851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca40ff760 0x7f9ca40ffb80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009910 tx=0x7f9c98009c20 comp rx=0 tx=0).stop 2024-04-27T03:55:30.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.115+0000 7f9cab0a9700 1 -- 172.21.15.46:0/1777890851 shutdown_connections 2024-04-27T03:55:30.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.115+0000 7f9cab0a9700 1 --2- 172.21.15.46:0/1777890851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca40ff760 0x7f9ca40ffb80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:30.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.115+0000 7f9cab0a9700 1 -- 172.21.15.46:0/1777890851 >> 172.21.15.46:0/1777890851 conn(0x7f9ca40fad00 msgr2=0x7f9ca40fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:30.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.115+0000 7f9cab0a9700 1 -- 172.21.15.46:0/1777890851 shutdown_connections 2024-04-27T03:55:30.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.115+0000 7f9cab0a9700 1 -- 172.21.15.46:0/1777890851 wait complete. 2024-04-27T03:55:30.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.116+0000 7f9cab0a9700 1 Processor -- start 2024-04-27T03:55:30.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.116+0000 7f9cab0a9700 1 -- start start 2024-04-27T03:55:30.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.116+0000 7f9cab0a9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca40ff760 0x7f9ca4195d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:30.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.116+0000 7f9cab0a9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca4196240 con 0x7f9ca40ff760 2024-04-27T03:55:30.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.116+0000 7f9ca8e45700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca40ff760 0x7f9ca4195d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:30.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.116+0000 7f9ca8e45700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca40ff760 0x7f9ca4195d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47060/0 (socket says 172.21.15.46:47060) 2024-04-27T03:55:30.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.116+0000 7f9ca8e45700 1 -- 172.21.15.46:0/1688450799 learned_addr learned my addr 172.21.15.46:0/1688450799 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:30.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.116+0000 7f9ca8e45700 1 -- 172.21.15.46:0/1688450799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c980095f0 con 0x7f9ca40ff760 2024-04-27T03:55:30.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.117+0000 7f9ca8e45700 1 --2- 172.21.15.46:0/1688450799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca40ff760 0x7f9ca4195d00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9c98006010 tx=0x7f9c980050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:30.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.117+0000 7f9ca1ffb700 1 -- 172.21.15.46:0/1688450799 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c9801d070 con 0x7f9ca40ff760 2024-04-27T03:55:30.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.117+0000 7f9ca1ffb700 1 -- 172.21.15.46:0/1688450799 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c9800be20 con 0x7f9ca40ff760 2024-04-27T03:55:30.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.117+0000 7f9cab0a9700 1 -- 172.21.15.46:0/1688450799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca4196440 con 0x7f9ca40ff760 2024-04-27T03:55:30.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.117+0000 7f9ca1ffb700 1 -- 172.21.15.46:0/1688450799 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c98022de0 con 0x7f9ca40ff760 2024-04-27T03:55:30.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.117+0000 7f9cab0a9700 1 -- 172.21.15.46:0/1688450799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca41968e0 con 0x7f9ca40ff760 2024-04-27T03:55:30.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.118+0000 7f9ca1ffb700 1 -- 172.21.15.46:0/1688450799 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f9c9800f460 con 0x7f9ca40ff760 2024-04-27T03:55:30.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.118+0000 7f9ca1ffb700 1 --2- 172.21.15.46:0/1688450799 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f9c94038820 0x7f9c9403ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:30.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.118+0000 7f9ca1ffb700 1 -- 172.21.15.46:0/1688450799 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9c9804cbc0 con 0x7f9ca40ff760 2024-04-27T03:55:30.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.118+0000 7f9cab0a9700 1 -- 172.21.15.46:0/1688450799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca418f860 con 0x7f9ca40ff760 2024-04-27T03:55:30.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.118+0000 7f9ca3fff700 1 --2- 172.21.15.46:0/1688450799 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f9c94038820 0x7f9c9403ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:30.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.119+0000 7f9ca3fff700 1 --2- 172.21.15.46:0/1688450799 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f9c94038820 0x7f9c9403ace0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9c90006fd0 tx=0x7f9c90006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:30.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.123+0000 7f9ca1ffb700 1 -- 172.21.15.46:0/1688450799 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f9c98027020 con 0x7f9ca40ff760 2024-04-27T03:55:30.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.332+0000 7f9cab0a9700 1 -- 172.21.15.46:0/1688450799 --> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ca4061320 con 0x7f9c94038820 2024-04-27T03:55:30.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.335+0000 7f9ca1ffb700 1 -- 172.21.15.46:0/1688450799 <== mgr.14162 v2:172.21.15.46:6800/3280787672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9ca4061320 con 0x7f9c94038820 2024-04-27T03:55:30.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.337+0000 7f9cab0a9700 1 -- 172.21.15.46:0/1688450799 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f9c94038820 msgr2=0x7f9c9403ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:30.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.337+0000 7f9cab0a9700 1 --2- 172.21.15.46:0/1688450799 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f9c94038820 0x7f9c9403ace0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9c90006fd0 tx=0x7f9c90006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:30.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.337+0000 7f9cab0a9700 1 -- 172.21.15.46:0/1688450799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca40ff760 msgr2=0x7f9ca4195d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:30.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.337+0000 7f9cab0a9700 1 --2- 172.21.15.46:0/1688450799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca40ff760 0x7f9ca4195d00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9c98006010 tx=0x7f9c980050b0 comp rx=0 tx=0).stop 2024-04-27T03:55:30.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.337+0000 7f9cab0a9700 1 -- 172.21.15.46:0/1688450799 shutdown_connections 2024-04-27T03:55:30.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.337+0000 7f9cab0a9700 1 --2- 172.21.15.46:0/1688450799 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f9c94038820 0x7f9c9403ace0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:30.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.337+0000 7f9cab0a9700 1 --2- 172.21.15.46:0/1688450799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca40ff760 0x7f9ca4195d00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:30.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.337+0000 7f9cab0a9700 1 -- 172.21.15.46:0/1688450799 >> 172.21.15.46:0/1688450799 conn(0x7f9ca40fad00 msgr2=0x7f9ca4069020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:30.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.337+0000 7f9cab0a9700 1 -- 172.21.15.46:0/1688450799 shutdown_connections 2024-04-27T03:55:30.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:30.337+0000 7f9cab0a9700 1 -- 172.21.15.46:0/1688450799 wait complete. 2024-04-27T03:55:31.154 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi073 2024-04-27T03:55:31.155 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-04-27T03:55:31.155 DEBUG:teuthology.orchestra.run.smithi073:> dd of=/etc/ceph/ceph.conf 2024-04-27T03:55:31.173 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-04-27T03:55:31.173 DEBUG:teuthology.orchestra.run.smithi073:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-27T03:55:31.230 INFO:tasks.cephadm:Adding host smithi073 to orchestrator... 2024-04-27T03:55:31.230 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph orch host add smithi073 2024-04-27T03:55:31.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:31 smithi046 ceph-mon[26899]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:55:31.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:31 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:31.700 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T03:55:32.891 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:32 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:32.892 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:32 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:32.892 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:32 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:32.892 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:32 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:32.892 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:32 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi046", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T03:55:32.892 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:32 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi046", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-27T03:55:32.892 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:32 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:55:32.892 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:32 smithi046 ceph-mon[26899]: Deploying daemon crash.smithi046 on smithi046 2024-04-27T03:55:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.387+0000 7f00d6995700 1 -- 172.21.15.46:0/3820459816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00ff770 msgr2=0x7f00d00ffb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.387+0000 7f00d6995700 1 --2- 172.21.15.46:0/3820459816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00ff770 0x7f00d00ffb90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f00c4009910 tx=0x7f00c4009c20 comp rx=0 tx=0).stop 2024-04-27T03:55:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.387+0000 7f00d6995700 1 -- 172.21.15.46:0/3820459816 shutdown_connections 2024-04-27T03:55:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.387+0000 7f00d6995700 1 --2- 172.21.15.46:0/3820459816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00ff770 0x7f00d00ffb90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.387+0000 7f00d6995700 1 -- 172.21.15.46:0/3820459816 >> 172.21.15.46:0/3820459816 conn(0x7f00d00fad10 msgr2=0x7f00d00fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.387+0000 7f00d6995700 1 -- 172.21.15.46:0/3820459816 shutdown_connections 2024-04-27T03:55:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.387+0000 7f00d6995700 1 -- 172.21.15.46:0/3820459816 wait complete. 2024-04-27T03:55:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.388+0000 7f00d6995700 1 Processor -- start 2024-04-27T03:55:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.388+0000 7f00d6995700 1 -- start start 2024-04-27T03:55:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.388+0000 7f00d6995700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00ff770 0x7f00d0195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.388+0000 7f00d6995700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d0196260 con 0x7f00d00ff770 2024-04-27T03:55:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.389+0000 7f00cffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00ff770 0x7f00d0195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.389+0000 7f00cffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00ff770 0x7f00d0195d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34804/0 (socket says 172.21.15.46:34804) 2024-04-27T03:55:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.389+0000 7f00cffff700 1 -- 172.21.15.46:0/3914896578 learned_addr learned my addr 172.21.15.46:0/3914896578 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.389+0000 7f00cffff700 1 -- 172.21.15.46:0/3914896578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c40095f0 con 0x7f00d00ff770 2024-04-27T03:55:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.389+0000 7f00cffff700 1 --2- 172.21.15.46:0/3914896578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00ff770 0x7f00d0195d20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f00c4006010 tx=0x7f00c4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.389+0000 7f00cd7fa700 1 -- 172.21.15.46:0/3914896578 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00c401d070 con 0x7f00d00ff770 2024-04-27T03:55:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.389+0000 7f00cd7fa700 1 -- 172.21.15.46:0/3914896578 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f00c4005dc0 con 0x7f00d00ff770 2024-04-27T03:55:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.389+0000 7f00cd7fa700 1 -- 172.21.15.46:0/3914896578 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00c4022e90 con 0x7f00d00ff770 2024-04-27T03:55:34.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.389+0000 7f00d6995700 1 -- 172.21.15.46:0/3914896578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d0196460 con 0x7f00d00ff770 2024-04-27T03:55:34.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.389+0000 7f00d6995700 1 -- 172.21.15.46:0/3914896578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d0196900 con 0x7f00d00ff770 2024-04-27T03:55:34.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.390+0000 7f00cd7fa700 1 -- 172.21.15.46:0/3914896578 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f00c400f4f0 con 0x7f00d00ff770 2024-04-27T03:55:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.390+0000 7f00d6995700 1 -- 172.21.15.46:0/3914896578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d0062510 con 0x7f00d00ff770 2024-04-27T03:55:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.390+0000 7f00cd7fa700 1 --2- 172.21.15.46:0/3914896578 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f00b8038760 0x7f00b803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.390+0000 7f00cd7fa700 1 -- 172.21.15.46:0/3914896578 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f00c404cb40 con 0x7f00d00ff770 2024-04-27T03:55:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.390+0000 7f00cf7fe700 1 --2- 172.21.15.46:0/3914896578 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f00b8038760 0x7f00b803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.391+0000 7f00cf7fe700 1 --2- 172.21.15.46:0/3914896578 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f00b8038760 0x7f00b803ac20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f00c0006fd0 tx=0x7f00c0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:34.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.395+0000 7f00cd7fa700 1 -- 172.21.15.46:0/3914896578 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f00c402a950 con 0x7f00d00ff770 2024-04-27T03:55:34.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:34.644+0000 7f00d6995700 1 -- 172.21.15.46:0/3914896578 --> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi073", "target": ["mon-mgr", ""]}) v1 -- 0x7f00d0068740 con 0x7f00b8038760 2024-04-27T03:55:35.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:35 smithi046 ceph-mon[26899]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:55:35.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:35 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:35.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:35 smithi046 ceph-mon[26899]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi073", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:55:35.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:35 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:35.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:35 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:35.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:35 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:35.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:35 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:36.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:36 smithi046 ceph-mon[26899]: Deploying daemon node-exporter.smithi046 on smithi046 2024-04-27T03:55:36.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:36 smithi046 ceph-mon[26899]: Deploying cephadm binary to smithi073 2024-04-27T03:55:37.162 INFO:teuthology.orchestra.run.smithi046.stdout:Added host 'smithi073' with addr '172.21.15.73' 2024-04-27T03:55:37.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:37.157+0000 7f00cd7fa700 1 -- 172.21.15.46:0/3914896578 <== mgr.14162 v2:172.21.15.46:6800/3280787672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f00d0068740 con 0x7f00b8038760 2024-04-27T03:55:37.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:37.159+0000 7f00d6995700 1 -- 172.21.15.46:0/3914896578 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f00b8038760 msgr2=0x7f00b803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:37.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:37.159+0000 7f00d6995700 1 --2- 172.21.15.46:0/3914896578 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f00b8038760 0x7f00b803ac20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f00c0006fd0 tx=0x7f00c0006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:37.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:37.159+0000 7f00d6995700 1 -- 172.21.15.46:0/3914896578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00ff770 msgr2=0x7f00d0195d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:37.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:37.159+0000 7f00d6995700 1 --2- 172.21.15.46:0/3914896578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00ff770 0x7f00d0195d20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f00c4006010 tx=0x7f00c4009f80 comp rx=0 tx=0).stop 2024-04-27T03:55:37.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:37.159+0000 7f00d6995700 1 -- 172.21.15.46:0/3914896578 shutdown_connections 2024-04-27T03:55:37.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:37.160+0000 7f00d6995700 1 --2- 172.21.15.46:0/3914896578 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f00b8038760 0x7f00b803ac20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:37.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:37.160+0000 7f00d6995700 1 --2- 172.21.15.46:0/3914896578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00ff770 0x7f00d0195d20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:37.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:37.160+0000 7f00d6995700 1 -- 172.21.15.46:0/3914896578 >> 172.21.15.46:0/3914896578 conn(0x7f00d00fad10 msgr2=0x7f00d0068030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:37.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:37.160+0000 7f00d6995700 1 -- 172.21.15.46:0/3914896578 shutdown_connections 2024-04-27T03:55:37.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:37.160+0000 7f00d6995700 1 -- 172.21.15.46:0/3914896578 wait complete. 2024-04-27T03:55:37.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:37 smithi046 ceph-mon[26899]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:55:38.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:38 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:38.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:38 smithi046 ceph-mon[26899]: Added host smithi073 2024-04-27T03:55:38.553 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-27T03:55:39.177 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T03:55:39.295 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:39 smithi046 ceph-mon[26899]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:55:39.295 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:39 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:40.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:40 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:40.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:40 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:40.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:40 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:40.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:40 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:40.529 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:40 smithi046 ceph-mon[26899]: Deploying daemon alertmanager.smithi046 on smithi046 2024-04-27T03:55:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.016+0000 7f985dce8700 1 -- 172.21.15.46:0/3519821778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9858101320 msgr2=0x7f9858103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.016+0000 7f985dce8700 1 --2- 172.21.15.46:0/3519821778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9858101320 0x7f9858103710 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9848009910 tx=0x7f9848009c20 comp rx=0 tx=0).stop 2024-04-27T03:55:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.016+0000 7f985dce8700 1 -- 172.21.15.46:0/3519821778 shutdown_connections 2024-04-27T03:55:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.016+0000 7f985dce8700 1 --2- 172.21.15.46:0/3519821778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9858101320 0x7f9858103710 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.016+0000 7f985dce8700 1 -- 172.21.15.46:0/3519821778 >> 172.21.15.46:0/3519821778 conn(0x7f98580fad30 msgr2=0x7f98580fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.016+0000 7f985dce8700 1 -- 172.21.15.46:0/3519821778 shutdown_connections 2024-04-27T03:55:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.016+0000 7f985dce8700 1 -- 172.21.15.46:0/3519821778 wait complete. 2024-04-27T03:55:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.017+0000 7f985dce8700 1 Processor -- start 2024-04-27T03:55:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.017+0000 7f985dce8700 1 -- start start 2024-04-27T03:55:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.017+0000 7f985dce8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9858101320 0x7f9858100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.017+0000 7f985dce8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9858101090 con 0x7f9858101320 2024-04-27T03:55:41.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.017+0000 7f98577fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9858101320 0x7f9858100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:41.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.017+0000 7f98577fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9858101320 0x7f9858100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47864/0 (socket says 172.21.15.46:47864) 2024-04-27T03:55:41.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.017+0000 7f98577fe700 1 -- 172.21.15.46:0/3366225404 learned_addr learned my addr 172.21.15.46:0/3366225404 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:41.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.017+0000 7f98577fe700 1 -- 172.21.15.46:0/3366225404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98480095f0 con 0x7f9858101320 2024-04-27T03:55:41.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.018+0000 7f98577fe700 1 --2- 172.21.15.46:0/3366225404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9858101320 0x7f9858100b50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9848006010 tx=0x7f98480050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:41.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.018+0000 7f9854ff9700 1 -- 172.21.15.46:0/3366225404 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f984801d070 con 0x7f9858101320 2024-04-27T03:55:41.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.018+0000 7f9854ff9700 1 -- 172.21.15.46:0/3366225404 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f984800be20 con 0x7f9858101320 2024-04-27T03:55:41.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.018+0000 7f985dce8700 1 -- 172.21.15.46:0/3366225404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98580ff2d0 con 0x7f9858101320 2024-04-27T03:55:41.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.018+0000 7f9854ff9700 1 -- 172.21.15.46:0/3366225404 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f984800f460 con 0x7f9858101320 2024-04-27T03:55:41.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.018+0000 7f985dce8700 1 -- 172.21.15.46:0/3366225404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98580ff770 con 0x7f9858101320 2024-04-27T03:55:41.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.019+0000 7f9854ff9700 1 -- 172.21.15.46:0/3366225404 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f984800f720 con 0x7f9858101320 2024-04-27T03:55:41.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.019+0000 7f985dce8700 1 -- 172.21.15.46:0/3366225404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9858106ff0 con 0x7f9858101320 2024-04-27T03:55:41.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.019+0000 7f9854ff9700 1 --2- 172.21.15.46:0/3366225404 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f98400387d0 0x7f984003ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:41.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.019+0000 7f9854ff9700 1 -- 172.21.15.46:0/3366225404 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f984804cb10 con 0x7f9858101320 2024-04-27T03:55:41.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.019+0000 7f9856ffd700 1 --2- 172.21.15.46:0/3366225404 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f98400387d0 0x7f984003ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:41.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.020+0000 7f9856ffd700 1 --2- 172.21.15.46:0/3366225404 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f98400387d0 0x7f984003ac90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f984c006fd0 tx=0x7f984c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:41.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.024+0000 7f9854ff9700 1 -- 172.21.15.46:0/3366225404 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f9848020550 con 0x7f9858101320 2024-04-27T03:55:41.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.234+0000 7f985dce8700 1 -- 172.21.15.46:0/3366225404 --> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98580ff9b0 con 0x7f98400387d0 2024-04-27T03:55:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.235+0000 7f9854ff9700 1 -- 172.21.15.46:0/3366225404 <== mgr.14162 v2:172.21.15.46:6800/3280787672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f98580ff9b0 con 0x7f98400387d0 2024-04-27T03:55:41.237 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T03:55:41.238 INFO:teuthology.orchestra.run.smithi046.stdout:[{"addr": "172.21.15.46", "hostname": "smithi046", "labels": [], "status": ""}, {"addr": "172.21.15.73", "hostname": "smithi073", "labels": [], "status": ""}] 2024-04-27T03:55:41.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.237+0000 7f985dce8700 1 -- 172.21.15.46:0/3366225404 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f98400387d0 msgr2=0x7f984003ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:41.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.237+0000 7f985dce8700 1 --2- 172.21.15.46:0/3366225404 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f98400387d0 0x7f984003ac90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f984c006fd0 tx=0x7f984c006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:41.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.237+0000 7f985dce8700 1 -- 172.21.15.46:0/3366225404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9858101320 msgr2=0x7f9858100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:41.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.237+0000 7f985dce8700 1 --2- 172.21.15.46:0/3366225404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9858101320 0x7f9858100b50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9848006010 tx=0x7f98480050b0 comp rx=0 tx=0).stop 2024-04-27T03:55:41.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.237+0000 7f985dce8700 1 -- 172.21.15.46:0/3366225404 shutdown_connections 2024-04-27T03:55:41.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.237+0000 7f985dce8700 1 --2- 172.21.15.46:0/3366225404 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f98400387d0 0x7f984003ac90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:41.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.237+0000 7f985dce8700 1 --2- 172.21.15.46:0/3366225404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9858101320 0x7f9858100b50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:41.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.237+0000 7f985dce8700 1 -- 172.21.15.46:0/3366225404 >> 172.21.15.46:0/3366225404 conn(0x7f98580fad30 msgr2=0x7f98580fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:41.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.237+0000 7f985dce8700 1 -- 172.21.15.46:0/3366225404 shutdown_connections 2024-04-27T03:55:41.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:41.237+0000 7f985dce8700 1 -- 172.21.15.46:0/3366225404 wait complete. 2024-04-27T03:55:41.583 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:41 smithi046 ceph-mon[26899]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:55:42.072 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-27T03:55:42.072 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-27T03:55:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T03:55:42.664 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:42 smithi046 ceph-mon[26899]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T03:55:43.729 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:43 smithi046 ceph-mon[26899]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:55:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.916+0000 7f2a07c5d700 1 -- 172.21.15.46:0/2605960416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a00100870 msgr2=0x7f2a00100c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.916+0000 7f2a07c5d700 1 --2- 172.21.15.46:0/2605960416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a00100870 0x7f2a00100c90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009910 tx=0x7f29fc009c20 comp rx=0 tx=0).stop 2024-04-27T03:55:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.916+0000 7f2a07c5d700 1 -- 172.21.15.46:0/2605960416 shutdown_connections 2024-04-27T03:55:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.916+0000 7f2a07c5d700 1 --2- 172.21.15.46:0/2605960416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a00100870 0x7f2a00100c90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.916+0000 7f2a07c5d700 1 -- 172.21.15.46:0/2605960416 >> 172.21.15.46:0/2605960416 conn(0x7f2a000fbe10 msgr2=0x7f2a000fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.916+0000 7f2a07c5d700 1 -- 172.21.15.46:0/2605960416 shutdown_connections 2024-04-27T03:55:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.916+0000 7f2a07c5d700 1 -- 172.21.15.46:0/2605960416 wait complete. 2024-04-27T03:55:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.917+0000 7f2a07c5d700 1 Processor -- start 2024-04-27T03:55:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.917+0000 7f2a07c5d700 1 -- start start 2024-04-27T03:55:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.917+0000 7f2a07c5d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a00100870 0x7f2a00195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.917+0000 7f2a07c5d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a00196250 con 0x7f2a00100870 2024-04-27T03:55:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.917+0000 7f2a059f9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a00100870 0x7f2a00195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.917+0000 7f2a059f9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a00100870 0x7f2a00195d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47880/0 (socket says 172.21.15.46:47880) 2024-04-27T03:55:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.917+0000 7f2a059f9700 1 -- 172.21.15.46:0/3094918746 learned_addr learned my addr 172.21.15.46:0/3094918746 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:55:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.917+0000 7f2a059f9700 1 -- 172.21.15.46:0/3094918746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc0095f0 con 0x7f2a00100870 2024-04-27T03:55:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.918+0000 7f2a059f9700 1 --2- 172.21.15.46:0/3094918746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a00100870 0x7f2a00195d10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f29fc006010 tx=0x7f29fc0050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.918+0000 7f29f6ffd700 1 -- 172.21.15.46:0/3094918746 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29fc01d070 con 0x7f2a00100870 2024-04-27T03:55:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.918+0000 7f2a07c5d700 1 -- 172.21.15.46:0/3094918746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a00196450 con 0x7f2a00100870 2024-04-27T03:55:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.918+0000 7f29f6ffd700 1 -- 172.21.15.46:0/3094918746 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29fc00be20 con 0x7f2a00100870 2024-04-27T03:55:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.918+0000 7f29f6ffd700 1 -- 172.21.15.46:0/3094918746 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29fc022de0 con 0x7f2a00100870 2024-04-27T03:55:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.918+0000 7f2a07c5d700 1 -- 172.21.15.46:0/3094918746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a001968f0 con 0x7f2a00100870 2024-04-27T03:55:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.919+0000 7f29f6ffd700 1 -- 172.21.15.46:0/3094918746 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f29fc0226c0 con 0x7f2a00100870 2024-04-27T03:55:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.919+0000 7f2a07c5d700 1 -- 172.21.15.46:0/3094918746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0018f6a0 con 0x7f2a00100870 2024-04-27T03:55:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.919+0000 7f29f6ffd700 1 --2- 172.21.15.46:0/3094918746 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f29ec0387b0 0x7f29ec03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.919+0000 7f29f6ffd700 1 -- 172.21.15.46:0/3094918746 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f29fc04c9a0 con 0x7f2a00100870 2024-04-27T03:55:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.919+0000 7f2a051f8700 1 --2- 172.21.15.46:0/3094918746 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f29ec0387b0 0x7f29ec03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.920+0000 7f2a051f8700 1 --2- 172.21.15.46:0/3094918746 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f29ec0387b0 0x7f29ec03ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f29f0006fd0 tx=0x7f29f0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:44.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:44.924+0000 7f29f6ffd700 1 -- 172.21.15.46:0/3094918746 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f29fc00b960 con 0x7f2a00100870 2024-04-27T03:55:45.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:45.139+0000 7f2a07c5d700 1 -- 172.21.15.46:0/3094918746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f2a00062510 con 0x7f2a00100870 2024-04-27T03:55:45.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:45 smithi046 ceph-mon[26899]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:55:45.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:45 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:46.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:46.127+0000 7f29f6ffd700 1 -- 172.21.15.46:0/3094918746 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2a00062510 con 0x7f2a00100870 2024-04-27T03:55:46.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:46.129+0000 7f2a07c5d700 1 -- 172.21.15.46:0/3094918746 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f29ec0387b0 msgr2=0x7f29ec03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:46.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:46.129+0000 7f2a07c5d700 1 --2- 172.21.15.46:0/3094918746 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f29ec0387b0 0x7f29ec03ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f29f0006fd0 tx=0x7f29f0006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:46.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:46.129+0000 7f2a07c5d700 1 -- 172.21.15.46:0/3094918746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a00100870 msgr2=0x7f2a00195d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:46.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:46.130+0000 7f2a07c5d700 1 --2- 172.21.15.46:0/3094918746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a00100870 0x7f2a00195d10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f29fc006010 tx=0x7f29fc0050b0 comp rx=0 tx=0).stop 2024-04-27T03:55:46.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:46.130+0000 7f2a07c5d700 1 -- 172.21.15.46:0/3094918746 shutdown_connections 2024-04-27T03:55:46.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:46.130+0000 7f2a07c5d700 1 --2- 172.21.15.46:0/3094918746 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f29ec0387b0 0x7f29ec03ac70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:46.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:46.130+0000 7f2a07c5d700 1 --2- 172.21.15.46:0/3094918746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a00100870 0x7f2a00195d10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:46.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:46.130+0000 7f2a07c5d700 1 -- 172.21.15.46:0/3094918746 >> 172.21.15.46:0/3094918746 conn(0x7f2a000fbe10 msgr2=0x7f2a000fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:46.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:46.130+0000 7f2a07c5d700 1 -- 172.21.15.46:0/3094918746 shutdown_connections 2024-04-27T03:55:46.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:55:46.130+0000 7f2a07c5d700 1 -- 172.21.15.46:0/3094918746 wait complete. 2024-04-27T03:55:46.133 INFO:teuthology.orchestra.run.smithi046.stderr:adjusted tunables profile to default 2024-04-27T03:55:46.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:46 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/3094918746' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-27T03:55:47.019 INFO:tasks.cephadm:Adding mon.smithi046 on smithi046 2024-04-27T03:55:47.019 INFO:tasks.cephadm:Adding mon.smithi073 on smithi073 2024-04-27T03:55:47.019 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph orch apply mon '2;smithi046:172.21.15.46=smithi046;smithi073:172.21.15.73=smithi073' 2024-04-27T03:55:47.266 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:55:47.312 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:55:47.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:47 smithi046 ceph-mon[26899]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:55:47.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:47 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/3094918746' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-27T03:55:47.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:47 smithi046 ceph-mon[26899]: osdmap e4: 0 total, 0 up, 0 in 2024-04-27T03:55:48.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.777+0000 7fbf4f077700 1 -- 172.21.15.73:0/204422612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf48101300 msgr2=0x7fbf481036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:48.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.777+0000 7fbf4f077700 1 --2- 172.21.15.73:0/204422612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf48101300 0x7fbf481036f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009910 tx=0x7fbf38009c20 comp rx=0 tx=0).stop 2024-04-27T03:55:48.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.777+0000 7fbf4f077700 1 -- 172.21.15.73:0/204422612 shutdown_connections 2024-04-27T03:55:48.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.777+0000 7fbf4f077700 1 --2- 172.21.15.73:0/204422612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf48101300 0x7fbf481036f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:48.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.777+0000 7fbf4f077700 1 -- 172.21.15.73:0/204422612 >> 172.21.15.73:0/204422612 conn(0x7fbf480fad10 msgr2=0x7fbf480fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:48.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.777+0000 7fbf4f077700 1 -- 172.21.15.73:0/204422612 shutdown_connections 2024-04-27T03:55:48.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.777+0000 7fbf4f077700 1 -- 172.21.15.73:0/204422612 wait complete. 2024-04-27T03:55:48.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.778+0000 7fbf4f077700 1 Processor -- start 2024-04-27T03:55:48.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.778+0000 7fbf4f077700 1 -- start start 2024-04-27T03:55:48.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.778+0000 7fbf4f077700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf48101300 0x7fbf48195cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:48.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.778+0000 7fbf4f077700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf481961f0 con 0x7fbf48101300 2024-04-27T03:55:48.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.778+0000 7fbf4ce13700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf48101300 0x7fbf48195cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:48.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.778+0000 7fbf4ce13700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf48101300 0x7fbf48195cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:42926/0 (socket says 172.21.15.73:42926) 2024-04-27T03:55:48.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.778+0000 7fbf4ce13700 1 -- 172.21.15.73:0/2347198815 learned_addr learned my addr 172.21.15.73:0/2347198815 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:55:48.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.779+0000 7fbf4ce13700 1 -- 172.21.15.73:0/2347198815 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf380095f0 con 0x7fbf48101300 2024-04-27T03:55:48.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.779+0000 7fbf4ce13700 1 --2- 172.21.15.73:0/2347198815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf48101300 0x7fbf48195cb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009ed0 tx=0x7fbf38004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:48.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.779+0000 7fbf45ffb700 1 -- 172.21.15.73:0/2347198815 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf3801d070 con 0x7fbf48101300 2024-04-27T03:55:48.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.779+0000 7fbf4f077700 1 -- 172.21.15.73:0/2347198815 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf481963f0 con 0x7fbf48101300 2024-04-27T03:55:48.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.779+0000 7fbf45ffb700 1 -- 172.21.15.73:0/2347198815 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf3800be20 con 0x7fbf48101300 2024-04-27T03:55:48.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.779+0000 7fbf45ffb700 1 -- 172.21.15.73:0/2347198815 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf3800f460 con 0x7fbf48101300 2024-04-27T03:55:48.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.779+0000 7fbf4f077700 1 -- 172.21.15.73:0/2347198815 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf48196890 con 0x7fbf48101300 2024-04-27T03:55:48.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.780+0000 7fbf45ffb700 1 -- 172.21.15.73:0/2347198815 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fbf3800f650 con 0x7fbf48101300 2024-04-27T03:55:48.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.780+0000 7fbf45ffb700 1 --2- 172.21.15.73:0/2347198815 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fbf300387d0 0x7fbf3003ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:48.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.780+0000 7fbf4f077700 1 -- 172.21.15.73:0/2347198815 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4818f6a0 con 0x7fbf48101300 2024-04-27T03:55:48.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.780+0000 7fbf45ffb700 1 -- 172.21.15.73:0/2347198815 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbf3804c980 con 0x7fbf48101300 2024-04-27T03:55:48.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.781+0000 7fbf47fff700 1 --2- 172.21.15.73:0/2347198815 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fbf300387d0 0x7fbf3003ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:48.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.781+0000 7fbf47fff700 1 --2- 172.21.15.73:0/2347198815 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fbf300387d0 0x7fbf3003ac90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c006fd0 tx=0x7fbf3c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:48.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.786+0000 7fbf45ffb700 1 -- 172.21.15.73:0/2347198815 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fbf3802c440 con 0x7fbf48101300 2024-04-27T03:55:48.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.992+0000 7fbf4f077700 1 -- 172.21.15.73:0/2347198815 --> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi046:172.21.15.46=smithi046;smithi073:172.21.15.73=smithi073", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf4802ce60 con 0x7fbf300387d0 2024-04-27T03:55:49.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:48.999+0000 7fbf45ffb700 1 -- 172.21.15.73:0/2347198815 <== mgr.14162 v2:172.21.15.46:6800/3280787672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fbf4802ce60 con 0x7fbf300387d0 2024-04-27T03:55:49.001 INFO:teuthology.orchestra.run.smithi073.stdout:Scheduled mon update... 2024-04-27T03:55:49.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:49.001+0000 7fbf4f077700 1 -- 172.21.15.73:0/2347198815 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fbf300387d0 msgr2=0x7fbf3003ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:49.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:49.001+0000 7fbf4f077700 1 --2- 172.21.15.73:0/2347198815 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fbf300387d0 0x7fbf3003ac90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c006fd0 tx=0x7fbf3c006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:49.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:49.001+0000 7fbf4f077700 1 -- 172.21.15.73:0/2347198815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf48101300 msgr2=0x7fbf48195cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:49.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:49.001+0000 7fbf4f077700 1 --2- 172.21.15.73:0/2347198815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf48101300 0x7fbf48195cb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009ed0 tx=0x7fbf38004690 comp rx=0 tx=0).stop 2024-04-27T03:55:49.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:49.001+0000 7fbf4f077700 1 -- 172.21.15.73:0/2347198815 shutdown_connections 2024-04-27T03:55:49.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:49.001+0000 7fbf4f077700 1 --2- 172.21.15.73:0/2347198815 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fbf300387d0 0x7fbf3003ac90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:49.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:49.001+0000 7fbf4f077700 1 --2- 172.21.15.73:0/2347198815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf48101300 0x7fbf48195cb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:49.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:49.001+0000 7fbf4f077700 1 -- 172.21.15.73:0/2347198815 >> 172.21.15.73:0/2347198815 conn(0x7fbf480fad10 msgr2=0x7fbf480fb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:49.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:49.001+0000 7fbf4f077700 1 -- 172.21.15.73:0/2347198815 shutdown_connections 2024-04-27T03:55:49.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:49.001+0000 7fbf4f077700 1 -- 172.21.15.73:0/2347198815 wait complete. 2024-04-27T03:55:49.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:49 smithi046 ceph-mon[26899]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:55:49.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:49 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:49.529 DEBUG:teuthology.orchestra.run.smithi073:mon.smithi073> sudo journalctl -f -n 0 -u ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@mon.smithi073.service 2024-04-27T03:55:49.532 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:55:49.532 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:55:49.762 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:55:49.809 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:55:50.383 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:50 smithi046 ceph-mon[26899]: from='client.14195 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi046:172.21.15.46=smithi046;smithi073:172.21.15.73=smithi073", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:55:50.383 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:50 smithi046 ceph-mon[26899]: Saving service mon spec with placement smithi046:172.21.15.46=smithi046;smithi073:172.21.15.73=smithi073;count:2 2024-04-27T03:55:51.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.288+0000 7fd504045700 1 -- 172.21.15.73:0/3788750159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc0ff770 msgr2=0x7fd4fc0ffb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:51.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.288+0000 7fd504045700 1 --2- 172.21.15.73:0/3788750159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc0ff770 0x7fd4fc0ffb90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd4f400b0a0 tx=0x7fd4f400b3b0 comp rx=0 tx=0).stop 2024-04-27T03:55:51.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.288+0000 7fd504045700 1 -- 172.21.15.73:0/3788750159 shutdown_connections 2024-04-27T03:55:51.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.288+0000 7fd504045700 1 --2- 172.21.15.73:0/3788750159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc0ff770 0x7fd4fc0ffb90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:51.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.288+0000 7fd504045700 1 -- 172.21.15.73:0/3788750159 >> 172.21.15.73:0/3788750159 conn(0x7fd4fc0fad10 msgr2=0x7fd4fc0fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:51.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.288+0000 7fd504045700 1 -- 172.21.15.73:0/3788750159 shutdown_connections 2024-04-27T03:55:51.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.288+0000 7fd504045700 1 -- 172.21.15.73:0/3788750159 wait complete. 2024-04-27T03:55:51.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.289+0000 7fd504045700 1 Processor -- start 2024-04-27T03:55:51.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.289+0000 7fd504045700 1 -- start start 2024-04-27T03:55:51.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.289+0000 7fd504045700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc0ff770 0x7fd4fc106240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:51.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.289+0000 7fd504045700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc106780 con 0x7fd4fc0ff770 2024-04-27T03:55:51.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.289+0000 7fd501de1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc0ff770 0x7fd4fc106240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:51.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.289+0000 7fd501de1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc0ff770 0x7fd4fc106240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:42956/0 (socket says 172.21.15.73:42956) 2024-04-27T03:55:51.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.289+0000 7fd501de1700 1 -- 172.21.15.73:0/587548394 learned_addr learned my addr 172.21.15.73:0/587548394 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:55:51.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.290+0000 7fd501de1700 1 -- 172.21.15.73:0/587548394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f4009d00 con 0x7fd4fc0ff770 2024-04-27T03:55:51.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.290+0000 7fd501de1700 1 --2- 172.21.15.73:0/587548394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc0ff770 0x7fd4fc106240 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd4f40062a0 tx=0x7fd4f400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:51.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.290+0000 7fd4f2ffd700 1 -- 172.21.15.73:0/587548394 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4f400e040 con 0x7fd4fc0ff770 2024-04-27T03:55:51.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.290+0000 7fd4f2ffd700 1 -- 172.21.15.73:0/587548394 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd4f4003d70 con 0x7fd4fc0ff770 2024-04-27T03:55:51.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.290+0000 7fd504045700 1 -- 172.21.15.73:0/587548394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4fc1049c0 con 0x7fd4fc0ff770 2024-04-27T03:55:51.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.290+0000 7fd4f2ffd700 1 -- 172.21.15.73:0/587548394 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4f401c890 con 0x7fd4fc0ff770 2024-04-27T03:55:51.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.290+0000 7fd504045700 1 -- 172.21.15.73:0/587548394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4fc104e60 con 0x7fd4fc0ff770 2024-04-27T03:55:51.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.291+0000 7fd4f2ffd700 1 -- 172.21.15.73:0/587548394 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fd4f401cb50 con 0x7fd4fc0ff770 2024-04-27T03:55:51.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.291+0000 7fd504045700 1 -- 172.21.15.73:0/587548394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4fc106980 con 0x7fd4fc0ff770 2024-04-27T03:55:51.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.291+0000 7fd4f2ffd700 1 --2- 172.21.15.73:0/587548394 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fd4e80387c0 0x7fd4e803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:51.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.291+0000 7fd4f2ffd700 1 -- 172.21.15.73:0/587548394 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd4f404c8e0 con 0x7fd4fc0ff770 2024-04-27T03:55:51.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.291+0000 7fd5015e0700 1 --2- 172.21.15.73:0/587548394 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fd4e80387c0 0x7fd4e803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:51.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.291+0000 7fd5015e0700 1 --2- 172.21.15.73:0/587548394 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fd4e80387c0 0x7fd4e803ac80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec006fd0 tx=0x7fd4ec006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:51.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.296+0000 7fd4f2ffd700 1 -- 172.21.15.73:0/587548394 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fd4f401a590 con 0x7fd4fc0ff770 2024-04-27T03:55:51.373 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:51 smithi046 ceph-mon[26899]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:55:51.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.563+0000 7fd504045700 1 -- 172.21.15.73:0/587548394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4fc106fd0 con 0x7fd4fc0ff770 2024-04-27T03:55:51.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.564+0000 7fd4f2ffd700 1 -- 172.21.15.73:0/587548394 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fd4f401a3b0 con 0x7fd4fc0ff770 2024-04-27T03:55:51.566 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:55:51.567 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:55:51.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.566+0000 7fd504045700 1 -- 172.21.15.73:0/587548394 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fd4e80387c0 msgr2=0x7fd4e803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:51.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.566+0000 7fd504045700 1 --2- 172.21.15.73:0/587548394 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fd4e80387c0 0x7fd4e803ac80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec006fd0 tx=0x7fd4ec006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:51.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.566+0000 7fd504045700 1 -- 172.21.15.73:0/587548394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc0ff770 msgr2=0x7fd4fc106240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:51.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.566+0000 7fd504045700 1 --2- 172.21.15.73:0/587548394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc0ff770 0x7fd4fc106240 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd4f40062a0 tx=0x7fd4f400bd40 comp rx=0 tx=0).stop 2024-04-27T03:55:51.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.566+0000 7fd504045700 1 -- 172.21.15.73:0/587548394 shutdown_connections 2024-04-27T03:55:51.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.566+0000 7fd504045700 1 --2- 172.21.15.73:0/587548394 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fd4e80387c0 0x7fd4e803ac80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:51.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.566+0000 7fd504045700 1 --2- 172.21.15.73:0/587548394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc0ff770 0x7fd4fc106240 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:51.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.566+0000 7fd504045700 1 -- 172.21.15.73:0/587548394 >> 172.21.15.73:0/587548394 conn(0x7fd4fc0fad10 msgr2=0x7fd4fc0fb860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:51.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.566+0000 7fd504045700 1 -- 172.21.15.73:0/587548394 shutdown_connections 2024-04-27T03:55:51.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:51.566+0000 7fd504045700 1 -- 172.21.15.73:0/587548394 wait complete. 2024-04-27T03:55:51.571 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:55:52.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:52 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/587548394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:55:53.076 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:55:53.076 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:55:53.278 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:55:53.326 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:55:53.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:53 smithi046 ceph-mon[26899]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:55:53.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:53 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:53.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:53 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:53.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:53 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:53.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:53 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:53.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:53 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:53.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:53 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:53.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:53 smithi046 ceph-mon[26899]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-27T03:55:53.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:53 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:53.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:53 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:53.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:53 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T03:55:53.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:53 smithi046 ceph-mon[26899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T03:55:53.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:53 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:53.677 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:53 smithi046 ceph-mon[26899]: Deploying daemon grafana.smithi046 on smithi046 2024-04-27T03:55:54.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.782+0000 7f4afadd9700 1 -- 172.21.15.73:0/3563133132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af40ff0f0 msgr2=0x7f4af4101520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:54.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.782+0000 7f4afadd9700 1 --2- 172.21.15.73:0/3563133132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af40ff0f0 0x7f4af4101520 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4aec00b0a0 tx=0x7f4aec00b3b0 comp rx=0 tx=0).stop 2024-04-27T03:55:54.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.782+0000 7f4afadd9700 1 -- 172.21.15.73:0/3563133132 shutdown_connections 2024-04-27T03:55:54.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.782+0000 7f4afadd9700 1 --2- 172.21.15.73:0/3563133132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af40ff0f0 0x7f4af4101520 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:54.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.782+0000 7f4afadd9700 1 -- 172.21.15.73:0/3563133132 >> 172.21.15.73:0/3563133132 conn(0x7f4af40fad10 msgr2=0x7f4af40fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:54.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.782+0000 7f4afadd9700 1 -- 172.21.15.73:0/3563133132 shutdown_connections 2024-04-27T03:55:54.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.782+0000 7f4afadd9700 1 -- 172.21.15.73:0/3563133132 wait complete. 2024-04-27T03:55:54.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.783+0000 7f4afadd9700 1 Processor -- start 2024-04-27T03:55:54.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.783+0000 7f4afadd9700 1 -- start start 2024-04-27T03:55:54.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.783+0000 7f4afadd9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af40ff0f0 0x7f4af419c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:54.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.783+0000 7f4afadd9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af419c840 con 0x7f4af40ff0f0 2024-04-27T03:55:54.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.783+0000 7f4af8b75700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af40ff0f0 0x7f4af419c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:54.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.783+0000 7f4af8b75700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af40ff0f0 0x7f4af419c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:55970/0 (socket says 172.21.15.73:55970) 2024-04-27T03:55:54.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.783+0000 7f4af8b75700 1 -- 172.21.15.73:0/1921091681 learned_addr learned my addr 172.21.15.73:0/1921091681 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:55:54.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.783+0000 7f4af8b75700 1 -- 172.21.15.73:0/1921091681 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aec009d00 con 0x7f4af40ff0f0 2024-04-27T03:55:54.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.783+0000 7f4af8b75700 1 --2- 172.21.15.73:0/1921091681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af40ff0f0 0x7f4af419c300 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4aec0062a0 tx=0x7f4aec00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:54.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.783+0000 7f4af1ffb700 1 -- 172.21.15.73:0/1921091681 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aec00e040 con 0x7f4af40ff0f0 2024-04-27T03:55:54.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.784+0000 7f4af1ffb700 1 -- 172.21.15.73:0/1921091681 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4aec003d70 con 0x7f4af40ff0f0 2024-04-27T03:55:54.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.784+0000 7f4afadd9700 1 -- 172.21.15.73:0/1921091681 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af419ca40 con 0x7f4af40ff0f0 2024-04-27T03:55:54.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.784+0000 7f4af1ffb700 1 -- 172.21.15.73:0/1921091681 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aec01c8d0 con 0x7f4af40ff0f0 2024-04-27T03:55:54.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.784+0000 7f4afadd9700 1 -- 172.21.15.73:0/1921091681 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af419cee0 con 0x7f4af40ff0f0 2024-04-27T03:55:54.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.785+0000 7f4af1ffb700 1 -- 172.21.15.73:0/1921091681 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f4aec01cb40 con 0x7f4af40ff0f0 2024-04-27T03:55:54.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.785+0000 7f4af1ffb700 1 --2- 172.21.15.73:0/1921091681 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f4adc038810 0x7f4adc03acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:54.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.785+0000 7f4af1ffb700 1 -- 172.21.15.73:0/1921091681 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4aec04ca50 con 0x7f4af40ff0f0 2024-04-27T03:55:54.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.785+0000 7f4afadd9700 1 -- 172.21.15.73:0/1921091681 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af41963e0 con 0x7f4af40ff0f0 2024-04-27T03:55:54.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.785+0000 7f4af3fff700 1 --2- 172.21.15.73:0/1921091681 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f4adc038810 0x7f4adc03acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:54.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.785+0000 7f4af3fff700 1 --2- 172.21.15.73:0/1921091681 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f4adc038810 0x7f4adc03acd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8006fd0 tx=0x7f4ae8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:54.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:54.790+0000 7f4af1ffb700 1 -- 172.21.15.73:0/1921091681 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f4aec003f90 con 0x7f4af40ff0f0 2024-04-27T03:55:55.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:55.058+0000 7f4afadd9700 1 -- 172.21.15.73:0/1921091681 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4af4062510 con 0x7f4af40ff0f0 2024-04-27T03:55:55.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:55.058+0000 7f4af1ffb700 1 -- 172.21.15.73:0/1921091681 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f4af4062510 con 0x7f4af40ff0f0 2024-04-27T03:55:55.061 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:55:55.061 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:55:55.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:55.060+0000 7f4afadd9700 1 -- 172.21.15.73:0/1921091681 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f4adc038810 msgr2=0x7f4adc03acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:55.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:55.060+0000 7f4afadd9700 1 --2- 172.21.15.73:0/1921091681 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f4adc038810 0x7f4adc03acd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8006fd0 tx=0x7f4ae8006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:55.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:55.060+0000 7f4afadd9700 1 -- 172.21.15.73:0/1921091681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af40ff0f0 msgr2=0x7f4af419c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:55.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:55.060+0000 7f4afadd9700 1 --2- 172.21.15.73:0/1921091681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af40ff0f0 0x7f4af419c300 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4aec0062a0 tx=0x7f4aec00bd40 comp rx=0 tx=0).stop 2024-04-27T03:55:55.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:55.060+0000 7f4afadd9700 1 -- 172.21.15.73:0/1921091681 shutdown_connections 2024-04-27T03:55:55.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:55.060+0000 7f4afadd9700 1 --2- 172.21.15.73:0/1921091681 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f4adc038810 0x7f4adc03acd0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:55.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:55.060+0000 7f4afadd9700 1 --2- 172.21.15.73:0/1921091681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af40ff0f0 0x7f4af419c300 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:55.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:55.061+0000 7f4afadd9700 1 -- 172.21.15.73:0/1921091681 >> 172.21.15.73:0/1921091681 conn(0x7f4af40fad10 msgr2=0x7f4af40fb600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:55.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:55.061+0000 7f4afadd9700 1 -- 172.21.15.73:0/1921091681 shutdown_connections 2024-04-27T03:55:55.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:55.061+0000 7f4afadd9700 1 -- 172.21.15.73:0/1921091681 wait complete. 2024-04-27T03:55:55.064 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:55:55.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:55 smithi046 ceph-mon[26899]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:55:55.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:55 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:55:55.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:55 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/1921091681' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:55:56.643 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:55:56.643 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:55:56.845 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:55:56.893 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:55:57.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:57 smithi046 ceph-mon[26899]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:55:58.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.395+0000 7f1399ec3700 1 -- 172.21.15.73:0/1061062356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1394100870 msgr2=0x7f1394100c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:58.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.395+0000 7f1399ec3700 1 --2- 172.21.15.73:0/1061062356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1394100870 0x7f1394100c90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f138400b0a0 tx=0x7f138400b3b0 comp rx=0 tx=0).stop 2024-04-27T03:55:58.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.395+0000 7f1399ec3700 1 -- 172.21.15.73:0/1061062356 shutdown_connections 2024-04-27T03:55:58.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.395+0000 7f1399ec3700 1 --2- 172.21.15.73:0/1061062356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1394100870 0x7f1394100c90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:58.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.395+0000 7f1399ec3700 1 -- 172.21.15.73:0/1061062356 >> 172.21.15.73:0/1061062356 conn(0x7f13940fbe30 msgr2=0x7f13940fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:58.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.395+0000 7f1399ec3700 1 -- 172.21.15.73:0/1061062356 shutdown_connections 2024-04-27T03:55:58.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.395+0000 7f1399ec3700 1 -- 172.21.15.73:0/1061062356 wait complete. 2024-04-27T03:55:58.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.396+0000 7f1399ec3700 1 Processor -- start 2024-04-27T03:55:58.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.396+0000 7f1399ec3700 1 -- start start 2024-04-27T03:55:58.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.396+0000 7f1399ec3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1394100870 0x7f1394195cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:58.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.396+0000 7f1399ec3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1394196230 con 0x7f1394100870 2024-04-27T03:55:58.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.396+0000 7f13937fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1394100870 0x7f1394195cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:58.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.396+0000 7f13937fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1394100870 0x7f1394195cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:55988/0 (socket says 172.21.15.73:55988) 2024-04-27T03:55:58.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.396+0000 7f13937fe700 1 -- 172.21.15.73:0/86589158 learned_addr learned my addr 172.21.15.73:0/86589158 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:55:58.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.396+0000 7f13937fe700 1 -- 172.21.15.73:0/86589158 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1384009d00 con 0x7f1394100870 2024-04-27T03:55:58.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.397+0000 7f13937fe700 1 --2- 172.21.15.73:0/86589158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1394100870 0x7f1394195cf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f138400bc90 tx=0x7f138400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:58.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.397+0000 7f1390ff9700 1 -- 172.21.15.73:0/86589158 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f138400e040 con 0x7f1394100870 2024-04-27T03:55:58.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.397+0000 7f1390ff9700 1 -- 172.21.15.73:0/86589158 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1384003d50 con 0x7f1394100870 2024-04-27T03:55:58.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.397+0000 7f1399ec3700 1 -- 172.21.15.73:0/86589158 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1394196430 con 0x7f1394100870 2024-04-27T03:55:58.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.397+0000 7f1390ff9700 1 -- 172.21.15.73:0/86589158 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f138401c8f0 con 0x7f1394100870 2024-04-27T03:55:58.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.397+0000 7f1399ec3700 1 -- 172.21.15.73:0/86589158 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13941968d0 con 0x7f1394100870 2024-04-27T03:55:58.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.397+0000 7f1390ff9700 1 -- 172.21.15.73:0/86589158 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f138401cbb0 con 0x7f1394100870 2024-04-27T03:55:58.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.397+0000 7f1399ec3700 1 -- 172.21.15.73:0/86589158 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f139418f6a0 con 0x7f1394100870 2024-04-27T03:55:58.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.398+0000 7f1390ff9700 1 --2- 172.21.15.73:0/86589158 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f137c038810 0x7f137c03acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:55:58.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.398+0000 7f1390ff9700 1 -- 172.21.15.73:0/86589158 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f138404ca40 con 0x7f1394100870 2024-04-27T03:55:58.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.398+0000 7f1392ffd700 1 --2- 172.21.15.73:0/86589158 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f137c038810 0x7f137c03acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:55:58.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.398+0000 7f1392ffd700 1 --2- 172.21.15.73:0/86589158 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f137c038810 0x7f137c03acd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1388006fd0 tx=0x7f1388006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:55:58.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.403+0000 7f1390ff9700 1 -- 172.21.15.73:0/86589158 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f138401a590 con 0x7f1394100870 2024-04-27T03:55:58.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.680+0000 7f1399ec3700 1 -- 172.21.15.73:0/86589158 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1394062510 con 0x7f1394100870 2024-04-27T03:55:58.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.680+0000 7f1390ff9700 1 -- 172.21.15.73:0/86589158 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f1394062510 con 0x7f1394100870 2024-04-27T03:55:58.683 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:55:58.683 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:55:58.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.682+0000 7f1399ec3700 1 -- 172.21.15.73:0/86589158 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f137c038810 msgr2=0x7f137c03acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:58.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.682+0000 7f1399ec3700 1 --2- 172.21.15.73:0/86589158 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f137c038810 0x7f137c03acd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1388006fd0 tx=0x7f1388006e40 comp rx=0 tx=0).stop 2024-04-27T03:55:58.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.682+0000 7f1399ec3700 1 -- 172.21.15.73:0/86589158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1394100870 msgr2=0x7f1394195cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:55:58.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.682+0000 7f1399ec3700 1 --2- 172.21.15.73:0/86589158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1394100870 0x7f1394195cf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f138400bc90 tx=0x7f138400bd70 comp rx=0 tx=0).stop 2024-04-27T03:55:58.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.682+0000 7f1399ec3700 1 -- 172.21.15.73:0/86589158 shutdown_connections 2024-04-27T03:55:58.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.682+0000 7f1399ec3700 1 --2- 172.21.15.73:0/86589158 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f137c038810 0x7f137c03acd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:58.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.682+0000 7f1399ec3700 1 --2- 172.21.15.73:0/86589158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1394100870 0x7f1394195cf0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:55:58.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.682+0000 7f1399ec3700 1 -- 172.21.15.73:0/86589158 >> 172.21.15.73:0/86589158 conn(0x7f13940fbe30 msgr2=0x7f13940fc890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:55:58.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.682+0000 7f1399ec3700 1 -- 172.21.15.73:0/86589158 shutdown_connections 2024-04-27T03:55:58.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:55:58.683+0000 7f1399ec3700 1 -- 172.21.15.73:0/86589158 wait complete. 2024-04-27T03:55:58.686 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:55:59.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:59 smithi046 ceph-mon[26899]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:55:59.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:55:59 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/86589158' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:56:00.358 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:56:00.358 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:56:00.558 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:00.605 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:01.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:01 smithi046 ceph-mon[26899]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:56:02.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.045+0000 7f84ea1d7700 1 -- 172.21.15.73:0/1782259504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e4100850 msgr2=0x7f84e4100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:02.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.045+0000 7f84ea1d7700 1 --2- 172.21.15.73:0/1782259504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e4100850 0x7f84e4100c70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009910 tx=0x7f84d8009c20 comp rx=0 tx=0).stop 2024-04-27T03:56:02.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.045+0000 7f84ea1d7700 1 -- 172.21.15.73:0/1782259504 shutdown_connections 2024-04-27T03:56:02.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.045+0000 7f84ea1d7700 1 --2- 172.21.15.73:0/1782259504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e4100850 0x7f84e4100c70 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:02.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.045+0000 7f84ea1d7700 1 -- 172.21.15.73:0/1782259504 >> 172.21.15.73:0/1782259504 conn(0x7f84e40fbdf0 msgr2=0x7f84e40fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:02.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.045+0000 7f84ea1d7700 1 -- 172.21.15.73:0/1782259504 shutdown_connections 2024-04-27T03:56:02.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.045+0000 7f84ea1d7700 1 -- 172.21.15.73:0/1782259504 wait complete. 2024-04-27T03:56:02.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.046+0000 7f84ea1d7700 1 Processor -- start 2024-04-27T03:56:02.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.046+0000 7f84ea1d7700 1 -- start start 2024-04-27T03:56:02.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.046+0000 7f84ea1d7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e4100850 0x7f84e4074b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:02.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.046+0000 7f84ea1d7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e4075090 con 0x7f84e4100850 2024-04-27T03:56:02.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.046+0000 7f84e37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e4100850 0x7f84e4074b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:02.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.046+0000 7f84e37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e4100850 0x7f84e4074b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:38254/0 (socket says 172.21.15.73:38254) 2024-04-27T03:56:02.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.046+0000 7f84e37fe700 1 -- 172.21.15.73:0/3949054150 learned_addr learned my addr 172.21.15.73:0/3949054150 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:56:02.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.046+0000 7f84e37fe700 1 -- 172.21.15.73:0/3949054150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d80095f0 con 0x7f84e4100850 2024-04-27T03:56:02.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.047+0000 7f84e37fe700 1 --2- 172.21.15.73:0/3949054150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e4100850 0x7f84e4074b50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f84d8006010 tx=0x7f84d8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:02.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.047+0000 7f84e0ff9700 1 -- 172.21.15.73:0/3949054150 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84d801d070 con 0x7f84e4100850 2024-04-27T03:56:02.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.047+0000 7f84e0ff9700 1 -- 172.21.15.73:0/3949054150 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84d8005dc0 con 0x7f84e4100850 2024-04-27T03:56:02.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.047+0000 7f84ea1d7700 1 -- 172.21.15.73:0/3949054150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e40732d0 con 0x7f84e4100850 2024-04-27T03:56:02.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.047+0000 7f84e0ff9700 1 -- 172.21.15.73:0/3949054150 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84d8022e90 con 0x7f84e4100850 2024-04-27T03:56:02.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.047+0000 7f84ea1d7700 1 -- 172.21.15.73:0/3949054150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e4073770 con 0x7f84e4100850 2024-04-27T03:56:02.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.048+0000 7f84e0ff9700 1 -- 172.21.15.73:0/3949054150 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f84d800b960 con 0x7f84e4100850 2024-04-27T03:56:02.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.048+0000 7f84ea1d7700 1 -- 172.21.15.73:0/3949054150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e4106fb0 con 0x7f84e4100850 2024-04-27T03:56:02.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.048+0000 7f84e0ff9700 1 --2- 172.21.15.73:0/3949054150 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f84cc038760 0x7f84cc03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:02.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.048+0000 7f84e0ff9700 1 -- 172.21.15.73:0/3949054150 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f84d804c8c0 con 0x7f84e4100850 2024-04-27T03:56:02.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.048+0000 7f84e2ffd700 1 --2- 172.21.15.73:0/3949054150 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f84cc038760 0x7f84cc03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:02.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.048+0000 7f84e2ffd700 1 --2- 172.21.15.73:0/3949054150 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f84cc038760 0x7f84cc03ac20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f84d4006fd0 tx=0x7f84d4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:02.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.053+0000 7f84e0ff9700 1 -- 172.21.15.73:0/3949054150 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f84d8022750 con 0x7f84e4100850 2024-04-27T03:56:02.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.323+0000 7f84ea1d7700 1 -- 172.21.15.73:0/3949054150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84e4062510 con 0x7f84e4100850 2024-04-27T03:56:02.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.323+0000 7f84e0ff9700 1 -- 172.21.15.73:0/3949054150 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f84d8027020 con 0x7f84e4100850 2024-04-27T03:56:02.326 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:56:02.326 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:56:02.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.325+0000 7f84ea1d7700 1 -- 172.21.15.73:0/3949054150 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f84cc038760 msgr2=0x7f84cc03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:02.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.325+0000 7f84ea1d7700 1 --2- 172.21.15.73:0/3949054150 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f84cc038760 0x7f84cc03ac20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f84d4006fd0 tx=0x7f84d4006e40 comp rx=0 tx=0).stop 2024-04-27T03:56:02.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.325+0000 7f84ea1d7700 1 -- 172.21.15.73:0/3949054150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e4100850 msgr2=0x7f84e4074b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:02.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.325+0000 7f84ea1d7700 1 --2- 172.21.15.73:0/3949054150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e4100850 0x7f84e4074b50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f84d8006010 tx=0x7f84d8009f80 comp rx=0 tx=0).stop 2024-04-27T03:56:02.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.325+0000 7f84ea1d7700 1 -- 172.21.15.73:0/3949054150 shutdown_connections 2024-04-27T03:56:02.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.325+0000 7f84ea1d7700 1 --2- 172.21.15.73:0/3949054150 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f84cc038760 0x7f84cc03ac20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:02.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.325+0000 7f84ea1d7700 1 --2- 172.21.15.73:0/3949054150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e4100850 0x7f84e4074b50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:02.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.325+0000 7f84ea1d7700 1 -- 172.21.15.73:0/3949054150 >> 172.21.15.73:0/3949054150 conn(0x7f84e40fbdf0 msgr2=0x7f84e40fc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:02.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.325+0000 7f84ea1d7700 1 -- 172.21.15.73:0/3949054150 shutdown_connections 2024-04-27T03:56:02.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:02.325+0000 7f84ea1d7700 1 -- 172.21.15.73:0/3949054150 wait complete. 2024-04-27T03:56:02.329 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:56:03.273 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:03 smithi046 ceph-mon[26899]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:56:03.273 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:03 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/3949054150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:56:03.907 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:56:03.907 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:56:04.108 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:04.155 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:05.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:05 smithi046 ceph-mon[26899]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:56:05.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.566+0000 7fe5d06ee700 1 -- 172.21.15.73:0/2868740224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c80ff770 msgr2=0x7fe5c80ffb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:05.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.566+0000 7fe5d06ee700 1 --2- 172.21.15.73:0/2868740224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c80ff770 0x7fe5c80ffb90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009910 tx=0x7fe5b8009c20 comp rx=0 tx=0).stop 2024-04-27T03:56:05.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.567+0000 7fe5d06ee700 1 -- 172.21.15.73:0/2868740224 shutdown_connections 2024-04-27T03:56:05.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.567+0000 7fe5d06ee700 1 --2- 172.21.15.73:0/2868740224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c80ff770 0x7fe5c80ffb90 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:05.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.567+0000 7fe5d06ee700 1 -- 172.21.15.73:0/2868740224 >> 172.21.15.73:0/2868740224 conn(0x7fe5c80fad10 msgr2=0x7fe5c80fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:05.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.567+0000 7fe5d06ee700 1 -- 172.21.15.73:0/2868740224 shutdown_connections 2024-04-27T03:56:05.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.567+0000 7fe5d06ee700 1 -- 172.21.15.73:0/2868740224 wait complete. 2024-04-27T03:56:05.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.567+0000 7fe5d06ee700 1 Processor -- start 2024-04-27T03:56:05.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.567+0000 7fe5d06ee700 1 -- start start 2024-04-27T03:56:05.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.567+0000 7fe5d06ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c80ff770 0x7fe5c8068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:05.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.567+0000 7fe5d06ee700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c8068fa0 con 0x7fe5c80ff770 2024-04-27T03:56:05.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.567+0000 7fe5ce48a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c80ff770 0x7fe5c8068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:05.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.568+0000 7fe5ce48a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c80ff770 0x7fe5c8068a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:38272/0 (socket says 172.21.15.73:38272) 2024-04-27T03:56:05.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.568+0000 7fe5ce48a700 1 -- 172.21.15.73:0/2013723872 learned_addr learned my addr 172.21.15.73:0/2013723872 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:56:05.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.568+0000 7fe5ce48a700 1 -- 172.21.15.73:0/2013723872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b80095f0 con 0x7fe5c80ff770 2024-04-27T03:56:05.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.568+0000 7fe5ce48a700 1 --2- 172.21.15.73:0/2013723872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c80ff770 0x7fe5c8068a60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8006010 tx=0x7fe5b8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:05.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.568+0000 7fe5bf7fe700 1 -- 172.21.15.73:0/2013723872 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5b801d070 con 0x7fe5c80ff770 2024-04-27T03:56:05.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.568+0000 7fe5bf7fe700 1 -- 172.21.15.73:0/2013723872 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5b8005dc0 con 0x7fe5c80ff770 2024-04-27T03:56:05.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.568+0000 7fe5d06ee700 1 -- 172.21.15.73:0/2013723872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c8198a60 con 0x7fe5c80ff770 2024-04-27T03:56:05.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.568+0000 7fe5bf7fe700 1 -- 172.21.15.73:0/2013723872 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5b8022e90 con 0x7fe5c80ff770 2024-04-27T03:56:05.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.568+0000 7fe5d06ee700 1 -- 172.21.15.73:0/2013723872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c8198ea0 con 0x7fe5c80ff770 2024-04-27T03:56:05.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.569+0000 7fe5bf7fe700 1 -- 172.21.15.73:0/2013723872 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fe5b800b960 con 0x7fe5c80ff770 2024-04-27T03:56:05.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.569+0000 7fe5d06ee700 1 -- 172.21.15.73:0/2013723872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c818e5a0 con 0x7fe5c80ff770 2024-04-27T03:56:05.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.569+0000 7fe5bf7fe700 1 --2- 172.21.15.73:0/2013723872 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fe5b40387c0 0x7fe5b403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:05.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.569+0000 7fe5bf7fe700 1 -- 172.21.15.73:0/2013723872 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe5b804c890 con 0x7fe5c80ff770 2024-04-27T03:56:05.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.569+0000 7fe5cdc89700 1 --2- 172.21.15.73:0/2013723872 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fe5b40387c0 0x7fe5b403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:05.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.570+0000 7fe5cdc89700 1 --2- 172.21.15.73:0/2013723872 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fe5b40387c0 0x7fe5b403ac80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4006fd0 tx=0x7fe5c4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:05.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.575+0000 7fe5bf7fe700 1 -- 172.21.15.73:0/2013723872 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fe5b80227a0 con 0x7fe5c80ff770 2024-04-27T03:56:05.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.843+0000 7fe5d06ee700 1 -- 172.21.15.73:0/2013723872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5c8062510 con 0x7fe5c80ff770 2024-04-27T03:56:05.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.847+0000 7fe5bf7fe700 1 -- 172.21.15.73:0/2013723872 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fe5b8027020 con 0x7fe5c80ff770 2024-04-27T03:56:05.849 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:56:05.849 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:56:05.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.849+0000 7fe5d06ee700 1 -- 172.21.15.73:0/2013723872 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fe5b40387c0 msgr2=0x7fe5b403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:05.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.849+0000 7fe5d06ee700 1 --2- 172.21.15.73:0/2013723872 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fe5b40387c0 0x7fe5b403ac80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4006fd0 tx=0x7fe5c4006e40 comp rx=0 tx=0).stop 2024-04-27T03:56:05.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.849+0000 7fe5d06ee700 1 -- 172.21.15.73:0/2013723872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c80ff770 msgr2=0x7fe5c8068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:05.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.849+0000 7fe5d06ee700 1 --2- 172.21.15.73:0/2013723872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c80ff770 0x7fe5c8068a60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8006010 tx=0x7fe5b8009f80 comp rx=0 tx=0).stop 2024-04-27T03:56:05.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.849+0000 7fe5d06ee700 1 -- 172.21.15.73:0/2013723872 shutdown_connections 2024-04-27T03:56:05.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.849+0000 7fe5d06ee700 1 --2- 172.21.15.73:0/2013723872 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7fe5b40387c0 0x7fe5b403ac80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:05.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.849+0000 7fe5d06ee700 1 --2- 172.21.15.73:0/2013723872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c80ff770 0x7fe5c8068a60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:05.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.849+0000 7fe5d06ee700 1 -- 172.21.15.73:0/2013723872 >> 172.21.15.73:0/2013723872 conn(0x7fe5c80fad10 msgr2=0x7fe5c80fb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:05.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.849+0000 7fe5d06ee700 1 -- 172.21.15.73:0/2013723872 shutdown_connections 2024-04-27T03:56:05.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:05.849+0000 7fe5d06ee700 1 -- 172.21.15.73:0/2013723872 wait complete. 2024-04-27T03:56:05.852 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:56:06.178 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:06 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/2013723872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:56:06.178 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:06 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:06.178 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:06 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:06.179 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:06 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:06.179 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:06 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:06.179 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:06 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:06.179 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:06 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:06.179 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:06 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:06.179 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:06 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:07.393 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:56:07.393 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:56:07.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:07 smithi046 ceph-mon[26899]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:56:07.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:07 smithi046 ceph-mon[26899]: Deploying daemon prometheus.smithi046 on smithi046 2024-04-27T03:56:07.592 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:07.649 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:09.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.079+0000 7f59f6f61700 1 -- 172.21.15.73:0/2608315733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f00ff760 msgr2=0x7f59f00ffb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:09.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.079+0000 7f59f6f61700 1 --2- 172.21.15.73:0/2608315733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f00ff760 0x7f59f00ffb80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f59e4009910 tx=0x7f59e4009c20 comp rx=0 tx=0).stop 2024-04-27T03:56:09.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.079+0000 7f59f6f61700 1 -- 172.21.15.73:0/2608315733 shutdown_connections 2024-04-27T03:56:09.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.079+0000 7f59f6f61700 1 --2- 172.21.15.73:0/2608315733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f00ff760 0x7f59f00ffb80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:09.082 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.079+0000 7f59f6f61700 1 -- 172.21.15.73:0/2608315733 >> 172.21.15.73:0/2608315733 conn(0x7f59f00fad00 msgr2=0x7f59f00fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:09.082 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.079+0000 7f59f6f61700 1 -- 172.21.15.73:0/2608315733 shutdown_connections 2024-04-27T03:56:09.082 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.079+0000 7f59f6f61700 1 -- 172.21.15.73:0/2608315733 wait complete. 2024-04-27T03:56:09.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.080+0000 7f59f6f61700 1 Processor -- start 2024-04-27T03:56:09.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.080+0000 7f59f6f61700 1 -- start start 2024-04-27T03:56:09.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.080+0000 7f59f6f61700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f00ff760 0x7f59f0195d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:09.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.080+0000 7f59f6f61700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f0196270 con 0x7f59f00ff760 2024-04-27T03:56:09.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.081+0000 7f59f4cfd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f00ff760 0x7f59f0195d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:09.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.081+0000 7f59f4cfd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f00ff760 0x7f59f0195d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:38294/0 (socket says 172.21.15.73:38294) 2024-04-27T03:56:09.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.081+0000 7f59f4cfd700 1 -- 172.21.15.73:0/2762109214 learned_addr learned my addr 172.21.15.73:0/2762109214 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:56:09.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.081+0000 7f59f4cfd700 1 -- 172.21.15.73:0/2762109214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e40095f0 con 0x7f59f00ff760 2024-04-27T03:56:09.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.081+0000 7f59f4cfd700 1 --2- 172.21.15.73:0/2762109214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f00ff760 0x7f59f0195d30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f59e4006010 tx=0x7f59e4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:09.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.081+0000 7f59edffb700 1 -- 172.21.15.73:0/2762109214 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59e401d070 con 0x7f59f00ff760 2024-04-27T03:56:09.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.081+0000 7f59edffb700 1 -- 172.21.15.73:0/2762109214 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59e4005dc0 con 0x7f59f00ff760 2024-04-27T03:56:09.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.081+0000 7f59f6f61700 1 -- 172.21.15.73:0/2762109214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f0196470 con 0x7f59f00ff760 2024-04-27T03:56:09.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.081+0000 7f59edffb700 1 -- 172.21.15.73:0/2762109214 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59e4022e90 con 0x7f59f00ff760 2024-04-27T03:56:09.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.081+0000 7f59f6f61700 1 -- 172.21.15.73:0/2762109214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f0196910 con 0x7f59f00ff760 2024-04-27T03:56:09.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.082+0000 7f59edffb700 1 -- 172.21.15.73:0/2762109214 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f59e400b960 con 0x7f59f00ff760 2024-04-27T03:56:09.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.082+0000 7f59f6f61700 1 -- 172.21.15.73:0/2762109214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f0062510 con 0x7f59f00ff760 2024-04-27T03:56:09.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.082+0000 7f59edffb700 1 --2- 172.21.15.73:0/2762109214 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f59e0038760 0x7f59e003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:09.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.082+0000 7f59edffb700 1 -- 172.21.15.73:0/2762109214 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f59e404c8c0 con 0x7f59f00ff760 2024-04-27T03:56:09.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.083+0000 7f59effff700 1 --2- 172.21.15.73:0/2762109214 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f59e0038760 0x7f59e003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:09.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.083+0000 7f59effff700 1 --2- 172.21.15.73:0/2762109214 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f59e0038760 0x7f59e003ac20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f59dc006fd0 tx=0x7f59dc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:09.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.088+0000 7f59edffb700 1 -- 172.21.15.73:0/2762109214 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f59e4022750 con 0x7f59f00ff760 2024-04-27T03:56:09.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.362+0000 7f59f6f61700 1 -- 172.21.15.73:0/2762109214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59f0105790 con 0x7f59f00ff760 2024-04-27T03:56:09.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.363+0000 7f59edffb700 1 -- 172.21.15.73:0/2762109214 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f59e4027020 con 0x7f59f00ff760 2024-04-27T03:56:09.365 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:56:09.365 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:56:09.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.365+0000 7f59f6f61700 1 -- 172.21.15.73:0/2762109214 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f59e0038760 msgr2=0x7f59e003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:09.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.365+0000 7f59f6f61700 1 --2- 172.21.15.73:0/2762109214 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f59e0038760 0x7f59e003ac20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f59dc006fd0 tx=0x7f59dc006e40 comp rx=0 tx=0).stop 2024-04-27T03:56:09.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.365+0000 7f59f6f61700 1 -- 172.21.15.73:0/2762109214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f00ff760 msgr2=0x7f59f0195d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:09.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.365+0000 7f59f6f61700 1 --2- 172.21.15.73:0/2762109214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f00ff760 0x7f59f0195d30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f59e4006010 tx=0x7f59e4009f80 comp rx=0 tx=0).stop 2024-04-27T03:56:09.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.365+0000 7f59f6f61700 1 -- 172.21.15.73:0/2762109214 shutdown_connections 2024-04-27T03:56:09.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.365+0000 7f59f6f61700 1 --2- 172.21.15.73:0/2762109214 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f59e0038760 0x7f59e003ac20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:09.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.365+0000 7f59f6f61700 1 --2- 172.21.15.73:0/2762109214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f00ff760 0x7f59f0195d30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:09.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.365+0000 7f59f6f61700 1 -- 172.21.15.73:0/2762109214 >> 172.21.15.73:0/2762109214 conn(0x7f59f00fad00 msgr2=0x7f59f0068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:09.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.365+0000 7f59f6f61700 1 -- 172.21.15.73:0/2762109214 shutdown_connections 2024-04-27T03:56:09.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:09.365+0000 7f59f6f61700 1 -- 172.21.15.73:0/2762109214 wait complete. 2024-04-27T03:56:09.368 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:56:09.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:09 smithi046 ceph-mon[26899]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:56:09.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:09 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:10.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:10 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/2762109214' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:56:10.973 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:56:10.973 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:56:11.172 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:11.219 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:11.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:11 smithi046 ceph-mon[26899]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:56:12.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.563+0000 7f1029684700 1 -- 172.21.15.73:0/3591847024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1024100850 msgr2=0x7f1024100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:12.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.563+0000 7f1029684700 1 --2- 172.21.15.73:0/3591847024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1024100850 0x7f1024100c70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1014009910 tx=0x7f1014009c20 comp rx=0 tx=0).stop 2024-04-27T03:56:12.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.563+0000 7f1029684700 1 -- 172.21.15.73:0/3591847024 shutdown_connections 2024-04-27T03:56:12.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.563+0000 7f1029684700 1 --2- 172.21.15.73:0/3591847024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1024100850 0x7f1024100c70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:12.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.563+0000 7f1029684700 1 -- 172.21.15.73:0/3591847024 >> 172.21.15.73:0/3591847024 conn(0x7f10240fbdf0 msgr2=0x7f10240fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:12.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.563+0000 7f1029684700 1 -- 172.21.15.73:0/3591847024 shutdown_connections 2024-04-27T03:56:12.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.563+0000 7f1029684700 1 -- 172.21.15.73:0/3591847024 wait complete. 2024-04-27T03:56:12.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.564+0000 7f1029684700 1 Processor -- start 2024-04-27T03:56:12.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.564+0000 7f1029684700 1 -- start start 2024-04-27T03:56:12.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.564+0000 7f1029684700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1024100850 0x7f1024195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:12.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.564+0000 7f1029684700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1024196260 con 0x7f1024100850 2024-04-27T03:56:12.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.564+0000 7f1022ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1024100850 0x7f1024195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:12.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.564+0000 7f1022ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1024100850 0x7f1024195d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:33582/0 (socket says 172.21.15.73:33582) 2024-04-27T03:56:12.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.564+0000 7f1022ffd700 1 -- 172.21.15.73:0/3292377414 learned_addr learned my addr 172.21.15.73:0/3292377414 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:56:12.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.564+0000 7f1022ffd700 1 -- 172.21.15.73:0/3292377414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10140095f0 con 0x7f1024100850 2024-04-27T03:56:12.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.564+0000 7f1022ffd700 1 --2- 172.21.15.73:0/3292377414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1024100850 0x7f1024195d20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1014006010 tx=0x7f1014009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:12.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.565+0000 7f1020ff9700 1 -- 172.21.15.73:0/3292377414 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f101401d070 con 0x7f1024100850 2024-04-27T03:56:12.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.565+0000 7f1020ff9700 1 -- 172.21.15.73:0/3292377414 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1014005dc0 con 0x7f1024100850 2024-04-27T03:56:12.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.565+0000 7f1029684700 1 -- 172.21.15.73:0/3292377414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1024196460 con 0x7f1024100850 2024-04-27T03:56:12.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.565+0000 7f1020ff9700 1 -- 172.21.15.73:0/3292377414 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1014022e90 con 0x7f1024100850 2024-04-27T03:56:12.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.565+0000 7f1029684700 1 -- 172.21.15.73:0/3292377414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1024196900 con 0x7f1024100850 2024-04-27T03:56:12.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.565+0000 7f1020ff9700 1 -- 172.21.15.73:0/3292377414 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f101400b960 con 0x7f1024100850 2024-04-27T03:56:12.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.566+0000 7f1029684700 1 -- 172.21.15.73:0/3292377414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102418f680 con 0x7f1024100850 2024-04-27T03:56:12.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.566+0000 7f1020ff9700 1 --2- 172.21.15.73:0/3292377414 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f10100387c0 0x7f101003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:12.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.566+0000 7f1020ff9700 1 -- 172.21.15.73:0/3292377414 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f101404c840 con 0x7f1024100850 2024-04-27T03:56:12.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.566+0000 7f101bfff700 1 --2- 172.21.15.73:0/3292377414 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f10100387c0 0x7f101003ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:12.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.566+0000 7f101bfff700 1 --2- 172.21.15.73:0/3292377414 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f10100387c0 0x7f101003ac80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f100c006fd0 tx=0x7f100c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:12.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.571+0000 7f1020ff9700 1 -- 172.21.15.73:0/3292377414 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f101400bd00 con 0x7f1024100850 2024-04-27T03:56:12.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.840+0000 7f1029684700 1 -- 172.21.15.73:0/3292377414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1024062510 con 0x7f1024100850 2024-04-27T03:56:12.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.840+0000 7f1020ff9700 1 -- 172.21.15.73:0/3292377414 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f1024062510 con 0x7f1024100850 2024-04-27T03:56:12.843 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:56:12.843 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:56:12.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.842+0000 7f1029684700 1 -- 172.21.15.73:0/3292377414 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f10100387c0 msgr2=0x7f101003ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:12.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.842+0000 7f1029684700 1 --2- 172.21.15.73:0/3292377414 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f10100387c0 0x7f101003ac80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f100c006fd0 tx=0x7f100c006e40 comp rx=0 tx=0).stop 2024-04-27T03:56:12.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.842+0000 7f1029684700 1 -- 172.21.15.73:0/3292377414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1024100850 msgr2=0x7f1024195d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:12.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.842+0000 7f1029684700 1 --2- 172.21.15.73:0/3292377414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1024100850 0x7f1024195d20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1014006010 tx=0x7f1014009f80 comp rx=0 tx=0).stop 2024-04-27T03:56:12.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.842+0000 7f1029684700 1 -- 172.21.15.73:0/3292377414 shutdown_connections 2024-04-27T03:56:12.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.842+0000 7f1029684700 1 --2- 172.21.15.73:0/3292377414 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f10100387c0 0x7f101003ac80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:12.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.842+0000 7f1029684700 1 --2- 172.21.15.73:0/3292377414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1024100850 0x7f1024195d20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:12.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.842+0000 7f1029684700 1 -- 172.21.15.73:0/3292377414 >> 172.21.15.73:0/3292377414 conn(0x7f10240fbdf0 msgr2=0x7f10240fc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:12.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.842+0000 7f1029684700 1 -- 172.21.15.73:0/3292377414 shutdown_connections 2024-04-27T03:56:12.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:12.842+0000 7f1029684700 1 -- 172.21.15.73:0/3292377414 wait complete. 2024-04-27T03:56:12.846 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:56:13.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:13 smithi046 ceph-mon[26899]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:56:13.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:13 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/3292377414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:56:14.426 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:56:14.427 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:56:14.627 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:14.673 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:15.344 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:15 smithi046 ceph-mon[26899]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:56:16.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.109+0000 7ff0bf778700 1 -- 172.21.15.73:0/1396253694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b80feb30 msgr2=0x7ff0b80fef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:16.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.109+0000 7ff0bf778700 1 --2- 172.21.15.73:0/1396253694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b80feb30 0x7ff0b80fef50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4009910 tx=0x7ff0b4009c20 comp rx=0 tx=0).stop 2024-04-27T03:56:16.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.109+0000 7ff0bf778700 1 -- 172.21.15.73:0/1396253694 shutdown_connections 2024-04-27T03:56:16.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.109+0000 7ff0bf778700 1 --2- 172.21.15.73:0/1396253694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b80feb30 0x7ff0b80fef50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:16.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.109+0000 7ff0bf778700 1 -- 172.21.15.73:0/1396253694 >> 172.21.15.73:0/1396253694 conn(0x7ff0b80fa490 msgr2=0x7ff0b80fc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:16.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.109+0000 7ff0bf778700 1 -- 172.21.15.73:0/1396253694 shutdown_connections 2024-04-27T03:56:16.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.109+0000 7ff0bf778700 1 -- 172.21.15.73:0/1396253694 wait complete. 2024-04-27T03:56:16.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.109+0000 7ff0bf778700 1 Processor -- start 2024-04-27T03:56:16.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.109+0000 7ff0bf778700 1 -- start start 2024-04-27T03:56:16.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.109+0000 7ff0bf778700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b80feb30 0x7ff0b8197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:16.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.110+0000 7ff0bf778700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b8198470 con 0x7ff0b80feb30 2024-04-27T03:56:16.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.110+0000 7ff0bd514700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b80feb30 0x7ff0b8197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:16.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.110+0000 7ff0bd514700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b80feb30 0x7ff0b8197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:33602/0 (socket says 172.21.15.73:33602) 2024-04-27T03:56:16.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.110+0000 7ff0bd514700 1 -- 172.21.15.73:0/2263575257 learned_addr learned my addr 172.21.15.73:0/2263575257 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:56:16.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.110+0000 7ff0bd514700 1 -- 172.21.15.73:0/2263575257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b40095f0 con 0x7ff0b80feb30 2024-04-27T03:56:16.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.110+0000 7ff0bd514700 1 --2- 172.21.15.73:0/2263575257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b80feb30 0x7ff0b8197f30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4006010 tx=0x7ff0b4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:16.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.110+0000 7ff0ae7fc700 1 -- 172.21.15.73:0/2263575257 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0b401d070 con 0x7ff0b80feb30 2024-04-27T03:56:16.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.110+0000 7ff0bf778700 1 -- 172.21.15.73:0/2263575257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b8198670 con 0x7ff0b80feb30 2024-04-27T03:56:16.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.110+0000 7ff0ae7fc700 1 -- 172.21.15.73:0/2263575257 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0b4005dc0 con 0x7ff0b80feb30 2024-04-27T03:56:16.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.110+0000 7ff0ae7fc700 1 -- 172.21.15.73:0/2263575257 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0b4022e90 con 0x7ff0b80feb30 2024-04-27T03:56:16.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.110+0000 7ff0bf778700 1 -- 172.21.15.73:0/2263575257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b8198b10 con 0x7ff0b80feb30 2024-04-27T03:56:16.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.111+0000 7ff0bf778700 1 -- 172.21.15.73:0/2263575257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b8191aa0 con 0x7ff0b80feb30 2024-04-27T03:56:16.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.111+0000 7ff0ae7fc700 1 -- 172.21.15.73:0/2263575257 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7ff0b4022470 con 0x7ff0b80feb30 2024-04-27T03:56:16.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.111+0000 7ff0ae7fc700 1 --2- 172.21.15.73:0/2263575257 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7ff0a4038760 0x7ff0a403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:16.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.112+0000 7ff0ae7fc700 1 -- 172.21.15.73:0/2263575257 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff0b404c9a0 con 0x7ff0b80feb30 2024-04-27T03:56:16.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.112+0000 7ff0bcd13700 1 --2- 172.21.15.73:0/2263575257 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7ff0a4038760 0x7ff0a403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:16.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.112+0000 7ff0bcd13700 1 --2- 172.21.15.73:0/2263575257 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7ff0a4038760 0x7ff0a403ac20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8006fd0 tx=0x7ff0a8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:16.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.117+0000 7ff0ae7fc700 1 -- 172.21.15.73:0/2263575257 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7ff0b400b960 con 0x7ff0b80feb30 2024-04-27T03:56:16.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.389+0000 7ff0bf778700 1 -- 172.21.15.73:0/2263575257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0b8062510 con 0x7ff0b80feb30 2024-04-27T03:56:16.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.389+0000 7ff0ae7fc700 1 -- 172.21.15.73:0/2263575257 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7ff0b8062510 con 0x7ff0b80feb30 2024-04-27T03:56:16.392 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:56:16.392 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:56:16.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.391+0000 7ff0bf778700 1 -- 172.21.15.73:0/2263575257 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7ff0a4038760 msgr2=0x7ff0a403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:16.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.391+0000 7ff0bf778700 1 --2- 172.21.15.73:0/2263575257 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7ff0a4038760 0x7ff0a403ac20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8006fd0 tx=0x7ff0a8006e40 comp rx=0 tx=0).stop 2024-04-27T03:56:16.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.391+0000 7ff0bf778700 1 -- 172.21.15.73:0/2263575257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b80feb30 msgr2=0x7ff0b8197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:16.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.391+0000 7ff0bf778700 1 --2- 172.21.15.73:0/2263575257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b80feb30 0x7ff0b8197f30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4006010 tx=0x7ff0b4009f80 comp rx=0 tx=0).stop 2024-04-27T03:56:16.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.391+0000 7ff0bf778700 1 -- 172.21.15.73:0/2263575257 shutdown_connections 2024-04-27T03:56:16.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.391+0000 7ff0bf778700 1 --2- 172.21.15.73:0/2263575257 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7ff0a4038760 0x7ff0a403ac20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:16.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.391+0000 7ff0bf778700 1 --2- 172.21.15.73:0/2263575257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b80feb30 0x7ff0b8197f30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:16.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.391+0000 7ff0bf778700 1 -- 172.21.15.73:0/2263575257 >> 172.21.15.73:0/2263575257 conn(0x7ff0b80fa490 msgr2=0x7ff0b80fc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:16.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.391+0000 7ff0bf778700 1 -- 172.21.15.73:0/2263575257 shutdown_connections 2024-04-27T03:56:16.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:16.391+0000 7ff0bf778700 1 -- 172.21.15.73:0/2263575257 wait complete. 2024-04-27T03:56:16.395 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:56:17.406 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:17 smithi046 ceph-mon[26899]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:56:17.406 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:17 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/2263575257' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:56:18.030 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:56:18.031 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:56:18.231 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:18.278 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:19.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:19 smithi046 ceph-mon[26899]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:56:19.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.571+0000 7ff1b91d4700 1 -- 172.21.15.73:0/633884824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1b40ff760 msgr2=0x7ff1b40ffb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:19.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.571+0000 7ff1b91d4700 1 --2- 172.21.15.73:0/633884824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1b40ff760 0x7ff1b40ffb80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009910 tx=0x7ff1a4009c20 comp rx=0 tx=0).stop 2024-04-27T03:56:19.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.571+0000 7ff1b91d4700 1 -- 172.21.15.73:0/633884824 shutdown_connections 2024-04-27T03:56:19.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.571+0000 7ff1b91d4700 1 --2- 172.21.15.73:0/633884824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1b40ff760 0x7ff1b40ffb80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:19.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.571+0000 7ff1b91d4700 1 -- 172.21.15.73:0/633884824 >> 172.21.15.73:0/633884824 conn(0x7ff1b40fad00 msgr2=0x7ff1b40fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:19.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.571+0000 7ff1b91d4700 1 -- 172.21.15.73:0/633884824 shutdown_connections 2024-04-27T03:56:19.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.571+0000 7ff1b91d4700 1 -- 172.21.15.73:0/633884824 wait complete. 2024-04-27T03:56:19.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.572+0000 7ff1b91d4700 1 Processor -- start 2024-04-27T03:56:19.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.572+0000 7ff1b91d4700 1 -- start start 2024-04-27T03:56:19.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.572+0000 7ff1b91d4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1b40ff760 0x7ff1b4195d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:19.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.572+0000 7ff1b91d4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b4196270 con 0x7ff1b40ff760 2024-04-27T03:56:19.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.572+0000 7ff1b2d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1b40ff760 0x7ff1b4195d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:19.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.572+0000 7ff1b2d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1b40ff760 0x7ff1b4195d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:33612/0 (socket says 172.21.15.73:33612) 2024-04-27T03:56:19.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.572+0000 7ff1b2d9d700 1 -- 172.21.15.73:0/872406574 learned_addr learned my addr 172.21.15.73:0/872406574 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:56:19.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.573+0000 7ff1b2d9d700 1 -- 172.21.15.73:0/872406574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a40095f0 con 0x7ff1b40ff760 2024-04-27T03:56:19.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.573+0000 7ff1b2d9d700 1 --2- 172.21.15.73:0/872406574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1b40ff760 0x7ff1b4195d30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4006010 tx=0x7ff1a4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:19.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.573+0000 7ff1b0d99700 1 -- 172.21.15.73:0/872406574 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1a401d070 con 0x7ff1b40ff760 2024-04-27T03:56:19.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.573+0000 7ff1b0d99700 1 -- 172.21.15.73:0/872406574 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1a4005dc0 con 0x7ff1b40ff760 2024-04-27T03:56:19.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.573+0000 7ff1b91d4700 1 -- 172.21.15.73:0/872406574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b4196470 con 0x7ff1b40ff760 2024-04-27T03:56:19.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.573+0000 7ff1b0d99700 1 -- 172.21.15.73:0/872406574 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1a4022e90 con 0x7ff1b40ff760 2024-04-27T03:56:19.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.573+0000 7ff1b91d4700 1 -- 172.21.15.73:0/872406574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b4196910 con 0x7ff1b40ff760 2024-04-27T03:56:19.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.574+0000 7ff1b0d99700 1 -- 172.21.15.73:0/872406574 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7ff1a4022470 con 0x7ff1b40ff760 2024-04-27T03:56:19.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.574+0000 7ff1b91d4700 1 -- 172.21.15.73:0/872406574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b4062510 con 0x7ff1b40ff760 2024-04-27T03:56:19.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.574+0000 7ff1b0d99700 1 --2- 172.21.15.73:0/872406574 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7ff1a0038760 0x7ff1a003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:19.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.574+0000 7ff1b0d99700 1 -- 172.21.15.73:0/872406574 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff1a404c8e0 con 0x7ff1b40ff760 2024-04-27T03:56:19.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.575+0000 7ff1aa59c700 1 --2- 172.21.15.73:0/872406574 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7ff1a0038760 0x7ff1a003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:19.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.575+0000 7ff1aa59c700 1 --2- 172.21.15.73:0/872406574 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7ff1a0038760 0x7ff1a003ac20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff19c006fd0 tx=0x7ff19c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:19.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.580+0000 7ff1b0d99700 1 -- 172.21.15.73:0/872406574 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7ff1a400b960 con 0x7ff1b40ff760 2024-04-27T03:56:19.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.851+0000 7ff1b91d4700 1 -- 172.21.15.73:0/872406574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1b4105790 con 0x7ff1b40ff760 2024-04-27T03:56:19.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.852+0000 7ff1b0d99700 1 -- 172.21.15.73:0/872406574 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7ff1a4027020 con 0x7ff1b40ff760 2024-04-27T03:56:19.854 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:56:19.855 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:56:19.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.854+0000 7ff1b91d4700 1 -- 172.21.15.73:0/872406574 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7ff1a0038760 msgr2=0x7ff1a003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:19.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.854+0000 7ff1b91d4700 1 --2- 172.21.15.73:0/872406574 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7ff1a0038760 0x7ff1a003ac20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff19c006fd0 tx=0x7ff19c006e40 comp rx=0 tx=0).stop 2024-04-27T03:56:19.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.854+0000 7ff1b91d4700 1 -- 172.21.15.73:0/872406574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1b40ff760 msgr2=0x7ff1b4195d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:19.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.854+0000 7ff1b91d4700 1 --2- 172.21.15.73:0/872406574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1b40ff760 0x7ff1b4195d30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4006010 tx=0x7ff1a4009f80 comp rx=0 tx=0).stop 2024-04-27T03:56:19.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.854+0000 7ff1b91d4700 1 -- 172.21.15.73:0/872406574 shutdown_connections 2024-04-27T03:56:19.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.854+0000 7ff1b91d4700 1 --2- 172.21.15.73:0/872406574 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7ff1a0038760 0x7ff1a003ac20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:19.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.854+0000 7ff1b91d4700 1 --2- 172.21.15.73:0/872406574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1b40ff760 0x7ff1b4195d30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:19.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.854+0000 7ff1b91d4700 1 -- 172.21.15.73:0/872406574 >> 172.21.15.73:0/872406574 conn(0x7ff1b40fad00 msgr2=0x7ff1b4068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:19.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.854+0000 7ff1b91d4700 1 -- 172.21.15.73:0/872406574 shutdown_connections 2024-04-27T03:56:19.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:19.854+0000 7ff1b91d4700 1 -- 172.21.15.73:0/872406574 wait complete. 2024-04-27T03:56:19.857 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:56:20.419 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:20 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/872406574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:56:21.350 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:56:21.350 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:56:21.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:21 smithi046 ceph-mon[26899]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:56:21.551 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:21.598 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:23.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.028+0000 7f961a0af700 1 -- 172.21.15.73:0/419878989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9614101310 msgr2=0x7f9614103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:23.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.028+0000 7f961a0af700 1 --2- 172.21.15.73:0/419878989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9614101310 0x7f9614103700 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9604009910 tx=0x7f9604009c20 comp rx=0 tx=0).stop 2024-04-27T03:56:23.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.028+0000 7f961a0af700 1 -- 172.21.15.73:0/419878989 shutdown_connections 2024-04-27T03:56:23.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.028+0000 7f961a0af700 1 --2- 172.21.15.73:0/419878989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9614101310 0x7f9614103700 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:23.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.028+0000 7f961a0af700 1 -- 172.21.15.73:0/419878989 >> 172.21.15.73:0/419878989 conn(0x7f96140fad40 msgr2=0x7f96140fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:23.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.028+0000 7f961a0af700 1 -- 172.21.15.73:0/419878989 shutdown_connections 2024-04-27T03:56:23.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.028+0000 7f961a0af700 1 -- 172.21.15.73:0/419878989 wait complete. 2024-04-27T03:56:23.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.028+0000 7f961a0af700 1 Processor -- start 2024-04-27T03:56:23.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.028+0000 7f961a0af700 1 -- start start 2024-04-27T03:56:23.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.029+0000 7f961a0af700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9614101310 0x7f9614195cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:23.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.029+0000 7f961a0af700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9614196200 con 0x7f9614101310 2024-04-27T03:56:23.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.029+0000 7f96137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9614101310 0x7f9614195cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:23.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.029+0000 7f96137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9614101310 0x7f9614195cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:32872/0 (socket says 172.21.15.73:32872) 2024-04-27T03:56:23.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.029+0000 7f96137fe700 1 -- 172.21.15.73:0/2653015946 learned_addr learned my addr 172.21.15.73:0/2653015946 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:56:23.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.029+0000 7f96137fe700 1 -- 172.21.15.73:0/2653015946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96040095f0 con 0x7f9614101310 2024-04-27T03:56:23.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.029+0000 7f96137fe700 1 --2- 172.21.15.73:0/2653015946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9614101310 0x7f9614195cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9604009ed0 tx=0x7f9604004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:23.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.029+0000 7f9610ff9700 1 -- 172.21.15.73:0/2653015946 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f960401d070 con 0x7f9614101310 2024-04-27T03:56:23.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.029+0000 7f961a0af700 1 -- 172.21.15.73:0/2653015946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9614196400 con 0x7f9614101310 2024-04-27T03:56:23.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.029+0000 7f9610ff9700 1 -- 172.21.15.73:0/2653015946 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f960400be20 con 0x7f9614101310 2024-04-27T03:56:23.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.029+0000 7f9610ff9700 1 -- 172.21.15.73:0/2653015946 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f960400f460 con 0x7f9614101310 2024-04-27T03:56:23.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.030+0000 7f961a0af700 1 -- 172.21.15.73:0/2653015946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96141968a0 con 0x7f9614101310 2024-04-27T03:56:23.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.030+0000 7f961a0af700 1 -- 172.21.15.73:0/2653015946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961418f6b0 con 0x7f9614101310 2024-04-27T03:56:23.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.030+0000 7f9610ff9700 1 -- 172.21.15.73:0/2653015946 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f9604022800 con 0x7f9614101310 2024-04-27T03:56:23.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.031+0000 7f9610ff9700 1 --2- 172.21.15.73:0/2653015946 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f95fc0387d0 0x7f95fc03ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:23.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.031+0000 7f9610ff9700 1 -- 172.21.15.73:0/2653015946 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f960404cc00 con 0x7f9614101310 2024-04-27T03:56:23.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.031+0000 7f9612ffd700 1 --2- 172.21.15.73:0/2653015946 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f95fc0387d0 0x7f95fc03ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:23.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.031+0000 7f9612ffd700 1 --2- 172.21.15.73:0/2653015946 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f95fc0387d0 0x7f95fc03ac90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9608006fd0 tx=0x7f9608006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:23.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.036+0000 7f9610ff9700 1 -- 172.21.15.73:0/2653015946 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f9604053080 con 0x7f9614101310 2024-04-27T03:56:23.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.303+0000 7f961a0af700 1 -- 172.21.15.73:0/2653015946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9614062510 con 0x7f9614101310 2024-04-27T03:56:23.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.303+0000 7f9610ff9700 1 -- 172.21.15.73:0/2653015946 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f9614062510 con 0x7f9614101310 2024-04-27T03:56:23.306 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:56:23.307 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:56:23.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.307+0000 7f961a0af700 1 -- 172.21.15.73:0/2653015946 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f95fc0387d0 msgr2=0x7f95fc03ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:23.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.307+0000 7f961a0af700 1 --2- 172.21.15.73:0/2653015946 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f95fc0387d0 0x7f95fc03ac90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9608006fd0 tx=0x7f9608006e40 comp rx=0 tx=0).stop 2024-04-27T03:56:23.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.307+0000 7f961a0af700 1 -- 172.21.15.73:0/2653015946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9614101310 msgr2=0x7f9614195cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:23.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.307+0000 7f961a0af700 1 --2- 172.21.15.73:0/2653015946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9614101310 0x7f9614195cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9604009ed0 tx=0x7f9604004690 comp rx=0 tx=0).stop 2024-04-27T03:56:23.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.307+0000 7f961a0af700 1 -- 172.21.15.73:0/2653015946 shutdown_connections 2024-04-27T03:56:23.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.307+0000 7f961a0af700 1 --2- 172.21.15.73:0/2653015946 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f95fc0387d0 0x7f95fc03ac90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:23.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.307+0000 7f961a0af700 1 --2- 172.21.15.73:0/2653015946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9614101310 0x7f9614195cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:23.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.307+0000 7f961a0af700 1 -- 172.21.15.73:0/2653015946 >> 172.21.15.73:0/2653015946 conn(0x7f96140fad40 msgr2=0x7f96140fb780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:23.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.307+0000 7f961a0af700 1 -- 172.21.15.73:0/2653015946 shutdown_connections 2024-04-27T03:56:23.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:23.307+0000 7f961a0af700 1 -- 172.21.15.73:0/2653015946 wait complete. 2024-04-27T03:56:23.310 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:56:23.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:23 smithi046 ceph-mon[26899]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:56:23.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:23 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:23.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:23 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:23.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:23 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:23.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:23 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-27T03:56:23.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:23 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/2653015946' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:56:24.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:24 smithi046 ceph-mon[26899]: from='mgr.14162 172.21.15.46:0/178807475' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-27T03:56:24.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:24 smithi046 ceph-mon[26899]: mgrmap e14: smithi046.qrfdvr(active, since 69s) 2024-04-27T03:56:24.955 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:56:24.956 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:56:25.149 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:25.196 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:26.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.454+0000 7f40c0e4c700 1 -- 172.21.15.73:0/509940724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40bc0ff760 msgr2=0x7f40bc0ffb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:26.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.454+0000 7f40c0e4c700 1 --2- 172.21.15.73:0/509940724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40bc0ff760 0x7f40bc0ffb80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f40ac009910 tx=0x7f40ac009c20 comp rx=0 tx=0).stop 2024-04-27T03:56:26.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.454+0000 7f40c0e4c700 1 -- 172.21.15.73:0/509940724 shutdown_connections 2024-04-27T03:56:26.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.454+0000 7f40c0e4c700 1 --2- 172.21.15.73:0/509940724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40bc0ff760 0x7f40bc0ffb80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:26.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.454+0000 7f40c0e4c700 1 -- 172.21.15.73:0/509940724 >> 172.21.15.73:0/509940724 conn(0x7f40bc0fad00 msgr2=0x7f40bc0fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:26.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.454+0000 7f40c0e4c700 1 -- 172.21.15.73:0/509940724 shutdown_connections 2024-04-27T03:56:26.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.454+0000 7f40c0e4c700 1 -- 172.21.15.73:0/509940724 wait complete. 2024-04-27T03:56:26.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.455+0000 7f40c0e4c700 1 Processor -- start 2024-04-27T03:56:26.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.455+0000 7f40c0e4c700 1 -- start start 2024-04-27T03:56:26.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.455+0000 7f40c0e4c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40bc0ff760 0x7f40bc197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:26.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.455+0000 7f40c0e4c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc198480 con 0x7f40bc0ff760 2024-04-27T03:56:26.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.455+0000 7f40ba59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40bc0ff760 0x7f40bc197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:26.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.455+0000 7f40ba59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40bc0ff760 0x7f40bc197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:32892/0 (socket says 172.21.15.73:32892) 2024-04-27T03:56:26.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.455+0000 7f40ba59c700 1 -- 172.21.15.73:0/2655737682 learned_addr learned my addr 172.21.15.73:0/2655737682 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:56:26.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.455+0000 7f40ba59c700 1 -- 172.21.15.73:0/2655737682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac0095f0 con 0x7f40bc0ff760 2024-04-27T03:56:26.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.456+0000 7f40ba59c700 1 --2- 172.21.15.73:0/2655737682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40bc0ff760 0x7f40bc197f40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f40ac006010 tx=0x7f40ac009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:26.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.456+0000 7f40b37fe700 1 -- 172.21.15.73:0/2655737682 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40ac01d070 con 0x7f40bc0ff760 2024-04-27T03:56:26.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.456+0000 7f40c0e4c700 1 -- 172.21.15.73:0/2655737682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40bc198680 con 0x7f40bc0ff760 2024-04-27T03:56:26.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.456+0000 7f40b37fe700 1 -- 172.21.15.73:0/2655737682 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f40ac005dc0 con 0x7f40bc0ff760 2024-04-27T03:56:26.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.456+0000 7f40b37fe700 1 -- 172.21.15.73:0/2655737682 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40ac022e90 con 0x7f40bc0ff760 2024-04-27T03:56:26.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.456+0000 7f40c0e4c700 1 -- 172.21.15.73:0/2655737682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40bc198b20 con 0x7f40bc0ff760 2024-04-27T03:56:26.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.457+0000 7f40b37fe700 1 -- 172.21.15.73:0/2655737682 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 14) v1 ==== 45626+0+0 (secure 0 0 0) 0x7f40ac00b960 con 0x7f40bc0ff760 2024-04-27T03:56:26.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.457+0000 7f40c0e4c700 1 -- 172.21.15.73:0/2655737682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40bc1920b0 con 0x7f40bc0ff760 2024-04-27T03:56:26.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.457+0000 7f40b37fe700 1 --2- 172.21.15.73:0/2655737682 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f40a80387b0 0x7f40a803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:26.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.457+0000 7f40b37fe700 1 -- 172.21.15.73:0/2655737682 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f40ac04ca80 con 0x7f40bc0ff760 2024-04-27T03:56:26.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.457+0000 7f40b9d9b700 1 -- 172.21.15.73:0/2655737682 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f40a80387b0 msgr2=0x7f40a803ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3280787672 2024-04-27T03:56:26.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.457+0000 7f40b9d9b700 1 --2- 172.21.15.73:0/2655737682 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f40a80387b0 0x7f40a803ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-27T03:56:26.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.462+0000 7f40b37fe700 1 -- 172.21.15.73:0/2655737682 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f40ac0227a0 con 0x7f40bc0ff760 2024-04-27T03:56:26.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.658+0000 7f40b9d9b700 1 -- 172.21.15.73:0/2655737682 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f40a80387b0 msgr2=0x7f40a803ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3280787672 2024-04-27T03:56:26.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.658+0000 7f40b9d9b700 1 --2- 172.21.15.73:0/2655737682 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f40a80387b0 0x7f40a803ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-27T03:56:26.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.734+0000 7f40c0e4c700 1 -- 172.21.15.73:0/2655737682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40bc062510 con 0x7f40bc0ff760 2024-04-27T03:56:26.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.734+0000 7f40b37fe700 1 -- 172.21.15.73:0/2655737682 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f40bc062510 con 0x7f40bc0ff760 2024-04-27T03:56:26.737 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:56:26.737 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:56:26.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.737+0000 7f40c0e4c700 1 -- 172.21.15.73:0/2655737682 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f40a80387b0 msgr2=0x7f40a803ac70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T03:56:26.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.737+0000 7f40c0e4c700 1 --2- 172.21.15.73:0/2655737682 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f40a80387b0 0x7f40a803ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:26.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.737+0000 7f40c0e4c700 1 -- 172.21.15.73:0/2655737682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40bc0ff760 msgr2=0x7f40bc197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:26.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.737+0000 7f40c0e4c700 1 --2- 172.21.15.73:0/2655737682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40bc0ff760 0x7f40bc197f40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f40ac006010 tx=0x7f40ac009f80 comp rx=0 tx=0).stop 2024-04-27T03:56:26.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.737+0000 7f40c0e4c700 1 -- 172.21.15.73:0/2655737682 shutdown_connections 2024-04-27T03:56:26.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.737+0000 7f40c0e4c700 1 --2- 172.21.15.73:0/2655737682 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f40a80387b0 0x7f40a803ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:26.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.737+0000 7f40c0e4c700 1 --2- 172.21.15.73:0/2655737682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40bc0ff760 0x7f40bc197f40 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:26.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.737+0000 7f40c0e4c700 1 -- 172.21.15.73:0/2655737682 >> 172.21.15.73:0/2655737682 conn(0x7f40bc0fad00 msgr2=0x7f40bc18f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:26.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.737+0000 7f40c0e4c700 1 -- 172.21.15.73:0/2655737682 shutdown_connections 2024-04-27T03:56:26.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:26.737+0000 7f40c0e4c700 1 -- 172.21.15.73:0/2655737682 wait complete. 2024-04-27T03:56:26.741 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:56:26.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:26 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/2655737682' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:56:28.366 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:56:28.367 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:56:28.567 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:28.614 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:30.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.195+0000 7f5d7874f700 1 -- 172.21.15.73:0/2794743889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d70101310 msgr2=0x7f5d70103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:30.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.195+0000 7f5d7874f700 1 --2- 172.21.15.73:0/2794743889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d70101310 0x7f5d70103700 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800b0a0 tx=0x7f5d6800b3b0 comp rx=0 tx=0).stop 2024-04-27T03:56:30.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.195+0000 7f5d7874f700 1 -- 172.21.15.73:0/2794743889 shutdown_connections 2024-04-27T03:56:30.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.195+0000 7f5d7874f700 1 --2- 172.21.15.73:0/2794743889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d70101310 0x7f5d70103700 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:30.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.195+0000 7f5d7874f700 1 -- 172.21.15.73:0/2794743889 >> 172.21.15.73:0/2794743889 conn(0x7f5d700fad00 msgr2=0x7f5d700fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:30.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.195+0000 7f5d7874f700 1 -- 172.21.15.73:0/2794743889 shutdown_connections 2024-04-27T03:56:30.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.195+0000 7f5d7874f700 1 -- 172.21.15.73:0/2794743889 wait complete. 2024-04-27T03:56:30.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.196+0000 7f5d7874f700 1 Processor -- start 2024-04-27T03:56:30.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.196+0000 7f5d7874f700 1 -- start start 2024-04-27T03:56:30.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.196+0000 7f5d7874f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d70101310 0x7f5d70195cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:30.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.196+0000 7f5d7874f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d70196200 con 0x7f5d70101310 2024-04-27T03:56:30.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.196+0000 7f5d764eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d70101310 0x7f5d70195cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:30.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.196+0000 7f5d764eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d70101310 0x7f5d70195cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:32896/0 (socket says 172.21.15.73:32896) 2024-04-27T03:56:30.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.196+0000 7f5d764eb700 1 -- 172.21.15.73:0/3030545940 learned_addr learned my addr 172.21.15.73:0/3030545940 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:56:30.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.196+0000 7f5d764eb700 1 -- 172.21.15.73:0/3030545940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d68009d00 con 0x7f5d70101310 2024-04-27T03:56:30.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.197+0000 7f5d764eb700 1 --2- 172.21.15.73:0/3030545940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d70101310 0x7f5d70195cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5d680062a0 tx=0x7f5d6800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:30.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.197+0000 7f5d677fe700 1 -- 172.21.15.73:0/3030545940 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d6800e040 con 0x7f5d70101310 2024-04-27T03:56:30.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.197+0000 7f5d7874f700 1 -- 172.21.15.73:0/3030545940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d70196400 con 0x7f5d70101310 2024-04-27T03:56:30.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.197+0000 7f5d677fe700 1 -- 172.21.15.73:0/3030545940 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d68003d70 con 0x7f5d70101310 2024-04-27T03:56:30.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.197+0000 7f5d677fe700 1 -- 172.21.15.73:0/3030545940 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d6801c8d0 con 0x7f5d70101310 2024-04-27T03:56:30.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.197+0000 7f5d7874f700 1 -- 172.21.15.73:0/3030545940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d701968a0 con 0x7f5d70101310 2024-04-27T03:56:30.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.198+0000 7f5d7874f700 1 -- 172.21.15.73:0/3030545940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7018f6a0 con 0x7f5d70101310 2024-04-27T03:56:30.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.198+0000 7f5d677fe700 1 -- 172.21.15.73:0/3030545940 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 14) v1 ==== 45626+0+0 (secure 0 0 0) 0x7f5d68012640 con 0x7f5d70101310 2024-04-27T03:56:30.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.198+0000 7f5d677fe700 1 --2- 172.21.15.73:0/3030545940 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f5d5c038810 0x7f5d5c03acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:30.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.198+0000 7f5d677fe700 1 -- 172.21.15.73:0/3030545940 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5d6804ca50 con 0x7f5d70101310 2024-04-27T03:56:30.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.198+0000 7f5d75cea700 1 -- 172.21.15.73:0/3030545940 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f5d5c038810 msgr2=0x7f5d5c03acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3280787672 2024-04-27T03:56:30.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.198+0000 7f5d75cea700 1 --2- 172.21.15.73:0/3030545940 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f5d5c038810 0x7f5d5c03acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-27T03:56:30.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.207+0000 7f5d677fe700 1 -- 172.21.15.73:0/3030545940 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f5d6801a590 con 0x7f5d70101310 2024-04-27T03:56:30.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.399+0000 7f5d75cea700 1 -- 172.21.15.73:0/3030545940 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f5d5c038810 msgr2=0x7f5d5c03acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3280787672 2024-04-27T03:56:30.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.399+0000 7f5d75cea700 1 --2- 172.21.15.73:0/3030545940 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f5d5c038810 0x7f5d5c03acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-27T03:56:30.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.477+0000 7f5d7874f700 1 -- 172.21.15.73:0/3030545940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d70062510 con 0x7f5d70101310 2024-04-27T03:56:30.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.478+0000 7f5d677fe700 1 -- 172.21.15.73:0/3030545940 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f5d6801a3b0 con 0x7f5d70101310 2024-04-27T03:56:30.481 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:56:30.481 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:56:30.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.480+0000 7f5d7874f700 1 -- 172.21.15.73:0/3030545940 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f5d5c038810 msgr2=0x7f5d5c03acd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T03:56:30.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.480+0000 7f5d7874f700 1 --2- 172.21.15.73:0/3030545940 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f5d5c038810 0x7f5d5c03acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:30.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.480+0000 7f5d7874f700 1 -- 172.21.15.73:0/3030545940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d70101310 msgr2=0x7f5d70195cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:30.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.480+0000 7f5d7874f700 1 --2- 172.21.15.73:0/3030545940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d70101310 0x7f5d70195cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5d680062a0 tx=0x7f5d6800bd40 comp rx=0 tx=0).stop 2024-04-27T03:56:30.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.480+0000 7f5d7874f700 1 -- 172.21.15.73:0/3030545940 shutdown_connections 2024-04-27T03:56:30.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.480+0000 7f5d7874f700 1 --2- 172.21.15.73:0/3030545940 >> [v2:172.21.15.46:6800/3280787672,v1:172.21.15.46:6801/3280787672] conn(0x7f5d5c038810 0x7f5d5c03acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:30.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.480+0000 7f5d7874f700 1 --2- 172.21.15.73:0/3030545940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d70101310 0x7f5d70195cc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:30.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.480+0000 7f5d7874f700 1 -- 172.21.15.73:0/3030545940 >> 172.21.15.73:0/3030545940 conn(0x7f5d700fad00 msgr2=0x7f5d700fb810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:30.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.480+0000 7f5d7874f700 1 -- 172.21.15.73:0/3030545940 shutdown_connections 2024-04-27T03:56:30.484 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:30.480+0000 7f5d7874f700 1 -- 172.21.15.73:0/3030545940 wait complete. 2024-04-27T03:56:30.484 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:56:30.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:30 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/3030545940' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:56:31.865 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:31 smithi046 ceph-mon[26899]: Active manager daemon smithi046.qrfdvr restarted 2024-04-27T03:56:31.865 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:31 smithi046 ceph-mon[26899]: Activating manager daemon smithi046.qrfdvr 2024-04-27T03:56:31.866 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:31 smithi046 ceph-mon[26899]: osdmap e5: 0 total, 0 up, 0 in 2024-04-27T03:56:31.866 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:31 smithi046 ceph-mon[26899]: mgrmap e15: smithi046.qrfdvr(active, starting, since 0.00328964s) 2024-04-27T03:56:31.866 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-04-27T03:56:31.866 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mgr metadata", "who": "smithi046.qrfdvr", "id": "smithi046.qrfdvr"}]: dispatch 2024-04-27T03:56:31.866 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T03:56:31.866 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T03:56:31.866 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T03:56:31.866 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:31 smithi046 ceph-mon[26899]: Manager daemon smithi046.qrfdvr is now available 2024-04-27T03:56:31.866 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:56:31.866 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:31.867 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:56:31.867 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.qrfdvr/mirror_snapshot_schedule"}]: dispatch 2024-04-27T03:56:31.867 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.qrfdvr/trash_purge_schedule"}]: dispatch 2024-04-27T03:56:31.867 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:32.104 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:56:32.105 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:56:32.293 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:32.343 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:32.809 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:32 smithi046 ceph-mon[26899]: [27/Apr/2024:03:56:31] ENGINE Bus STARTING 2024-04-27T03:56:32.809 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:32 smithi046 ceph-mon[26899]: [27/Apr/2024:03:56:31] ENGINE Serving on http://172.21.15.46:8765 2024-04-27T03:56:32.809 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:32 smithi046 ceph-mon[26899]: [27/Apr/2024:03:56:31] ENGINE Serving on https://172.21.15.46:7150 2024-04-27T03:56:32.809 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:32 smithi046 ceph-mon[26899]: [27/Apr/2024:03:56:31] ENGINE Bus STARTED 2024-04-27T03:56:32.809 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:32 smithi046 ceph-mon[26899]: mgrmap e16: smithi046.qrfdvr(active, since 1.00564s) 2024-04-27T03:56:32.809 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:32 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:33.915 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:33 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:34.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.319+0000 7f21df68e700 1 -- 172.21.15.73:0/3632773526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d8101320 msgr2=0x7f21d8103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:34.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.319+0000 7f21df68e700 1 --2- 172.21.15.73:0/3632773526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d8101320 0x7f21d8103710 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f21d4009910 tx=0x7f21d4009c20 comp rx=0 tx=0).stop 2024-04-27T03:56:34.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.319+0000 7f21df68e700 1 -- 172.21.15.73:0/3632773526 shutdown_connections 2024-04-27T03:56:34.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.319+0000 7f21df68e700 1 --2- 172.21.15.73:0/3632773526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d8101320 0x7f21d8103710 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:34.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.319+0000 7f21df68e700 1 -- 172.21.15.73:0/3632773526 >> 172.21.15.73:0/3632773526 conn(0x7f21d80fad30 msgr2=0x7f21d80fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:34.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.319+0000 7f21df68e700 1 -- 172.21.15.73:0/3632773526 shutdown_connections 2024-04-27T03:56:34.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.319+0000 7f21df68e700 1 -- 172.21.15.73:0/3632773526 wait complete. 2024-04-27T03:56:34.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.319+0000 7f21df68e700 1 Processor -- start 2024-04-27T03:56:34.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.319+0000 7f21df68e700 1 -- start start 2024-04-27T03:56:34.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.320+0000 7f21df68e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d8101320 0x7f21d8195d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:34.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.320+0000 7f21df68e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d8196240 con 0x7f21d8101320 2024-04-27T03:56:34.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.320+0000 7f21dd42a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d8101320 0x7f21d8195d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:34.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.320+0000 7f21dd42a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d8101320 0x7f21d8195d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:43280/0 (socket says 172.21.15.73:43280) 2024-04-27T03:56:34.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.320+0000 7f21dd42a700 1 -- 172.21.15.73:0/3435064460 learned_addr learned my addr 172.21.15.73:0/3435064460 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:56:34.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.320+0000 7f21dd42a700 1 -- 172.21.15.73:0/3435064460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d40095f0 con 0x7f21d8101320 2024-04-27T03:56:34.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.320+0000 7f21dd42a700 1 --2- 172.21.15.73:0/3435064460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d8101320 0x7f21d8195d00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f21d4009ed0 tx=0x7f21d4004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:34.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.321+0000 7f21ce7fc700 1 -- 172.21.15.73:0/3435064460 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21d401d070 con 0x7f21d8101320 2024-04-27T03:56:34.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.321+0000 7f21df68e700 1 -- 172.21.15.73:0/3435064460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d8196440 con 0x7f21d8101320 2024-04-27T03:56:34.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.321+0000 7f21ce7fc700 1 -- 172.21.15.73:0/3435064460 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21d400be20 con 0x7f21d8101320 2024-04-27T03:56:34.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.321+0000 7f21ce7fc700 1 -- 172.21.15.73:0/3435064460 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21d400f460 con 0x7f21d8101320 2024-04-27T03:56:34.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.321+0000 7f21df68e700 1 -- 172.21.15.73:0/3435064460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d81968e0 con 0x7f21d8101320 2024-04-27T03:56:34.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.321+0000 7f21df68e700 1 -- 172.21.15.73:0/3435064460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d818f6c0 con 0x7f21d8101320 2024-04-27T03:56:34.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.321+0000 7f21ce7fc700 1 -- 172.21.15.73:0/3435064460 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7f21d400b960 con 0x7f21d8101320 2024-04-27T03:56:34.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.322+0000 7f21ce7fc700 1 --2- 172.21.15.73:0/3435064460 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f21c40388c0 0x7f21c403ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:34.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.322+0000 7f21ce7fc700 1 -- 172.21.15.73:0/3435064460 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f21d404cb90 con 0x7f21d8101320 2024-04-27T03:56:34.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.322+0000 7f21dcc29700 1 --2- 172.21.15.73:0/3435064460 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f21c40388c0 0x7f21c403ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:34.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.322+0000 7f21dcc29700 1 --2- 172.21.15.73:0/3435064460 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f21c40388c0 0x7f21c403ad80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f21c8006fd0 tx=0x7f21c8006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:34.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.327+0000 7f21ce7fc700 1 -- 172.21.15.73:0/3435064460 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f21d402a540 con 0x7f21d8101320 2024-04-27T03:56:34.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.603+0000 7f21df68e700 1 -- 172.21.15.73:0/3435064460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21d8062510 con 0x7f21d8101320 2024-04-27T03:56:34.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.604+0000 7f21ce7fc700 1 -- 172.21.15.73:0/3435064460 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f21d8062510 con 0x7f21d8101320 2024-04-27T03:56:34.607 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:56:34.607 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:56:34.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.606+0000 7f21df68e700 1 -- 172.21.15.73:0/3435064460 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f21c40388c0 msgr2=0x7f21c403ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:34.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.606+0000 7f21df68e700 1 --2- 172.21.15.73:0/3435064460 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f21c40388c0 0x7f21c403ad80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f21c8006fd0 tx=0x7f21c8006e40 comp rx=0 tx=0).stop 2024-04-27T03:56:34.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.606+0000 7f21df68e700 1 -- 172.21.15.73:0/3435064460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d8101320 msgr2=0x7f21d8195d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:34.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.606+0000 7f21df68e700 1 --2- 172.21.15.73:0/3435064460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d8101320 0x7f21d8195d00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f21d4009ed0 tx=0x7f21d4004690 comp rx=0 tx=0).stop 2024-04-27T03:56:34.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.606+0000 7f21df68e700 1 -- 172.21.15.73:0/3435064460 shutdown_connections 2024-04-27T03:56:34.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.606+0000 7f21df68e700 1 --2- 172.21.15.73:0/3435064460 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f21c40388c0 0x7f21c403ad80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:34.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.606+0000 7f21df68e700 1 --2- 172.21.15.73:0/3435064460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d8101320 0x7f21d8195d00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:34.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.606+0000 7f21df68e700 1 -- 172.21.15.73:0/3435064460 >> 172.21.15.73:0/3435064460 conn(0x7f21d80fad30 msgr2=0x7f21d80fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:34.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.606+0000 7f21df68e700 1 -- 172.21.15.73:0/3435064460 shutdown_connections 2024-04-27T03:56:34.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:34.606+0000 7f21df68e700 1 -- 172.21.15.73:0/3435064460 wait complete. 2024-04-27T03:56:34.609 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:56:34.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:34 smithi046 ceph-mon[26899]: mgrmap e17: smithi046.qrfdvr(active, since 2s) 2024-04-27T03:56:34.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:34 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/3435064460' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:56:36.180 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:56:36.180 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:56:36.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:36 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:36.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:36 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:36.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:36 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:36.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:36 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:36.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:36 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi046", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:56:36.706 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:36.754 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T03:56:38.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:38.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:38.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:38.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:38.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi073", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:56:38.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:56:38.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:56:38.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: Updating smithi046:/etc/ceph/ceph.conf 2024-04-27T03:56:38.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: Updating smithi073:/etc/ceph/ceph.conf 2024-04-27T03:56:38.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: Updating smithi073:/var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T03:56:38.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: Updating smithi046:/var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T03:56:38.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: Updating smithi073:/etc/ceph/ceph.client.admin.keyring 2024-04-27T03:56:38.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2024-04-27T03:56:38.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: Updating smithi073:/var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T03:56:38.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: Updating smithi046:/var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T03:56:38.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:38.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:38.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:38.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:38.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:38.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi073", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-27T03:56:38.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi073", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-27T03:56:38.177 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:56:38.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.380+0000 7fbcb2f07700 1 -- 172.21.15.73:0/3821347032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcac101740 msgr2=0x7fbcac0fed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:38.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.380+0000 7fbcb2f07700 1 --2- 172.21.15.73:0/3821347032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcac101740 0x7fbcac0fed30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbca0009910 tx=0x7fbca0009c20 comp rx=0 tx=0).stop 2024-04-27T03:56:38.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.380+0000 7fbcb2f07700 1 -- 172.21.15.73:0/3821347032 shutdown_connections 2024-04-27T03:56:38.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.380+0000 7fbcb2f07700 1 --2- 172.21.15.73:0/3821347032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcac101740 0x7fbcac0fed30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:38.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.380+0000 7fbcb2f07700 1 -- 172.21.15.73:0/3821347032 >> 172.21.15.73:0/3821347032 conn(0x7fbcac0fa990 msgr2=0x7fbcac0fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:38.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.380+0000 7fbcb2f07700 1 -- 172.21.15.73:0/3821347032 shutdown_connections 2024-04-27T03:56:38.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.380+0000 7fbcb2f07700 1 -- 172.21.15.73:0/3821347032 wait complete. 2024-04-27T03:56:38.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.380+0000 7fbcb2f07700 1 Processor -- start 2024-04-27T03:56:38.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.381+0000 7fbcb2f07700 1 -- start start 2024-04-27T03:56:38.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.381+0000 7fbcb2f07700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcac101740 0x7fbcac1982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:38.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.381+0000 7fbcb2f07700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac198830 con 0x7fbcac101740 2024-04-27T03:56:38.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.381+0000 7fbcb0ca3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcac101740 0x7fbcac1982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:38.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.381+0000 7fbcb0ca3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcac101740 0x7fbcac1982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:43290/0 (socket says 172.21.15.73:43290) 2024-04-27T03:56:38.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.381+0000 7fbcb0ca3700 1 -- 172.21.15.73:0/523360656 learned_addr learned my addr 172.21.15.73:0/523360656 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:56:38.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.381+0000 7fbcb0ca3700 1 -- 172.21.15.73:0/523360656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca00095f0 con 0x7fbcac101740 2024-04-27T03:56:38.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.382+0000 7fbcb0ca3700 1 --2- 172.21.15.73:0/523360656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcac101740 0x7fbcac1982f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbca0005490 tx=0x7fbca0005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:38.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.382+0000 7fbca9ffb700 1 -- 172.21.15.73:0/523360656 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbca001d070 con 0x7fbcac101740 2024-04-27T03:56:38.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.382+0000 7fbcb2f07700 1 -- 172.21.15.73:0/523360656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcac198a30 con 0x7fbcac101740 2024-04-27T03:56:38.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.382+0000 7fbca9ffb700 1 -- 172.21.15.73:0/523360656 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbca0005700 con 0x7fbcac101740 2024-04-27T03:56:38.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.382+0000 7fbca9ffb700 1 -- 172.21.15.73:0/523360656 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbca0003950 con 0x7fbcac101740 2024-04-27T03:56:38.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.382+0000 7fbcb2f07700 1 -- 172.21.15.73:0/523360656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcac198e50 con 0x7fbcac101740 2024-04-27T03:56:38.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.383+0000 7fbcb2f07700 1 -- 172.21.15.73:0/523360656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcac0fff50 con 0x7fbcac101740 2024-04-27T03:56:38.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.383+0000 7fbca9ffb700 1 -- 172.21.15.73:0/523360656 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7fbca0004240 con 0x7fbcac101740 2024-04-27T03:56:38.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.383+0000 7fbca9ffb700 1 --2- 172.21.15.73:0/523360656 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbc9c0388c0 0x7fbc9c03ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:38.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.383+0000 7fbca9ffb700 1 -- 172.21.15.73:0/523360656 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fbca004c5a0 con 0x7fbcac101740 2024-04-27T03:56:38.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.383+0000 7fbcabfff700 1 --2- 172.21.15.73:0/523360656 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbc9c0388c0 0x7fbc9c03ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:38.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.384+0000 7fbcabfff700 1 --2- 172.21.15.73:0/523360656 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbc9c0388c0 0x7fbc9c03ad80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbc98006fd0 tx=0x7fbc98006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:38.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.389+0000 7fbca9ffb700 1 -- 172.21.15.73:0/523360656 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fbca000bc10 con 0x7fbcac101740 2024-04-27T03:56:38.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.672+0000 7fbcb2f07700 1 -- 172.21.15.73:0/523360656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbcac10ce60 con 0x7fbcac101740 2024-04-27T03:56:38.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.673+0000 7fbca9ffb700 1 -- 172.21.15.73:0/523360656 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fbca0026020 con 0x7fbcac101740 2024-04-27T03:56:38.677 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:56:38.677 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:56:38.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.675+0000 7fbcb2f07700 1 -- 172.21.15.73:0/523360656 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbc9c0388c0 msgr2=0x7fbc9c03ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:38.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.675+0000 7fbcb2f07700 1 --2- 172.21.15.73:0/523360656 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbc9c0388c0 0x7fbc9c03ad80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbc98006fd0 tx=0x7fbc98006e40 comp rx=0 tx=0).stop 2024-04-27T03:56:38.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.675+0000 7fbcb2f07700 1 -- 172.21.15.73:0/523360656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcac101740 msgr2=0x7fbcac1982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:38.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.675+0000 7fbcb2f07700 1 --2- 172.21.15.73:0/523360656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcac101740 0x7fbcac1982f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbca0005490 tx=0x7fbca0005570 comp rx=0 tx=0).stop 2024-04-27T03:56:38.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.675+0000 7fbcb2f07700 1 -- 172.21.15.73:0/523360656 shutdown_connections 2024-04-27T03:56:38.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.675+0000 7fbcb2f07700 1 --2- 172.21.15.73:0/523360656 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbc9c0388c0 0x7fbc9c03ad80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:38.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.675+0000 7fbcb2f07700 1 --2- 172.21.15.73:0/523360656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcac101740 0x7fbcac1982f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:38.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.675+0000 7fbcb2f07700 1 -- 172.21.15.73:0/523360656 >> 172.21.15.73:0/523360656 conn(0x7fbcac0fa990 msgr2=0x7fbcac0fba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:38.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.675+0000 7fbcb2f07700 1 -- 172.21.15.73:0/523360656 shutdown_connections 2024-04-27T03:56:38.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:38.675+0000 7fbcb2f07700 1 -- 172.21.15.73:0/523360656 wait complete. 2024-04-27T03:56:38.679 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:56:39.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:38 smithi046 ceph-mon[26899]: Deploying daemon ceph-exporter.smithi073 on smithi073 2024-04-27T03:56:39.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:38 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/523360656' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:56:40.978 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:56:40.978 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:56:41.258 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T03:56:42.617 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.615+0000 7f4c37d3d700 1 -- 172.21.15.73:0/4156003612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c30106800 msgr2=0x7f4c30106be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:42.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.615+0000 7f4c37d3d700 1 --2- 172.21.15.73:0/4156003612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c30106800 0x7f4c30106be0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c009910 tx=0x7f4c2c009c20 comp rx=0 tx=0).stop 2024-04-27T03:56:42.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.615+0000 7f4c37d3d700 1 -- 172.21.15.73:0/4156003612 shutdown_connections 2024-04-27T03:56:42.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.615+0000 7f4c37d3d700 1 --2- 172.21.15.73:0/4156003612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c30106800 0x7f4c30106be0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:42.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.615+0000 7f4c37d3d700 1 -- 172.21.15.73:0/4156003612 >> 172.21.15.73:0/4156003612 conn(0x7f4c30075290 msgr2=0x7f4c300756a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:42.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.615+0000 7f4c37d3d700 1 -- 172.21.15.73:0/4156003612 shutdown_connections 2024-04-27T03:56:42.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.615+0000 7f4c37d3d700 1 -- 172.21.15.73:0/4156003612 wait complete. 2024-04-27T03:56:42.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.616+0000 7f4c37d3d700 1 Processor -- start 2024-04-27T03:56:42.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.616+0000 7f4c37d3d700 1 -- start start 2024-04-27T03:56:42.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.616+0000 7f4c37d3d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c30106800 0x7f4c30196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:42.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.616+0000 7f4c37d3d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c301965d0 con 0x7f4c30106800 2024-04-27T03:56:42.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.616+0000 7f4c35ad9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c30106800 0x7f4c30196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:42.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.616+0000 7f4c35ad9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c30106800 0x7f4c30196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:34384/0 (socket says 172.21.15.73:34384) 2024-04-27T03:56:42.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.616+0000 7f4c35ad9700 1 -- 172.21.15.73:0/4246269502 learned_addr learned my addr 172.21.15.73:0/4246269502 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:56:42.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.616+0000 7f4c35ad9700 1 -- 172.21.15.73:0/4246269502 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c0095f0 con 0x7f4c30106800 2024-04-27T03:56:42.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.617+0000 7f4c35ad9700 1 --2- 172.21.15.73:0/4246269502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c30106800 0x7f4c30196090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c005490 tx=0x7f4c2c005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:42.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.617+0000 7f4c26ffd700 1 -- 172.21.15.73:0/4246269502 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c2c01d070 con 0x7f4c30106800 2024-04-27T03:56:42.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.617+0000 7f4c37d3d700 1 -- 172.21.15.73:0/4246269502 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c30196830 con 0x7f4c30106800 2024-04-27T03:56:42.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.617+0000 7f4c26ffd700 1 -- 172.21.15.73:0/4246269502 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c2c005700 con 0x7f4c30106800 2024-04-27T03:56:42.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.617+0000 7f4c26ffd700 1 -- 172.21.15.73:0/4246269502 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c2c0038e0 con 0x7f4c30106800 2024-04-27T03:56:42.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.617+0000 7f4c37d3d700 1 -- 172.21.15.73:0/4246269502 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c30196cb0 con 0x7f4c30106800 2024-04-27T03:56:42.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.618+0000 7f4c26ffd700 1 -- 172.21.15.73:0/4246269502 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7f4c2c003b50 con 0x7f4c30106800 2024-04-27T03:56:42.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.618+0000 7f4c37d3d700 1 -- 172.21.15.73:0/4246269502 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c300feed0 con 0x7f4c30106800 2024-04-27T03:56:42.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.618+0000 7f4c26ffd700 1 --2- 172.21.15.73:0/4246269502 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f4c1c038910 0x7f4c1c03add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:42.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.618+0000 7f4c26ffd700 1 -- 172.21.15.73:0/4246269502 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f4c2c04c8c0 con 0x7f4c30106800 2024-04-27T03:56:42.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.618+0000 7f4c352d8700 1 --2- 172.21.15.73:0/4246269502 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f4c1c038910 0x7f4c1c03add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:42.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.619+0000 7f4c352d8700 1 --2- 172.21.15.73:0/4246269502 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f4c1c038910 0x7f4c1c03add0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4c20006fd0 tx=0x7f4c20006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:42.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.623+0000 7f4c26ffd700 1 -- 172.21.15.73:0/4246269502 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f4c2c029950 con 0x7f4c30106800 2024-04-27T03:56:42.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:42 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:42.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:42 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:42.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:42 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:42.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:42 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:42.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:42 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T03:56:42.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:42 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-27T03:56:42.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:42 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:56:42.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:42 smithi046 ceph-mon[26899]: Deploying daemon crash.smithi073 on smithi073 2024-04-27T03:56:42.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.913+0000 7f4c37d3d700 1 -- 172.21.15.73:0/4246269502 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c30101f40 con 0x7f4c30106800 2024-04-27T03:56:42.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.914+0000 7f4c26ffd700 1 -- 172.21.15.73:0/4246269502 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f4c2c026070 con 0x7f4c30106800 2024-04-27T03:56:42.916 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:56:42.917 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:56:42.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.916+0000 7f4c37d3d700 1 -- 172.21.15.73:0/4246269502 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f4c1c038910 msgr2=0x7f4c1c03add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:42.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.916+0000 7f4c37d3d700 1 --2- 172.21.15.73:0/4246269502 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f4c1c038910 0x7f4c1c03add0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4c20006fd0 tx=0x7f4c20006e40 comp rx=0 tx=0).stop 2024-04-27T03:56:42.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.916+0000 7f4c37d3d700 1 -- 172.21.15.73:0/4246269502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c30106800 msgr2=0x7f4c30196090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:42.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.916+0000 7f4c37d3d700 1 --2- 172.21.15.73:0/4246269502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c30106800 0x7f4c30196090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c005490 tx=0x7f4c2c005570 comp rx=0 tx=0).stop 2024-04-27T03:56:42.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.916+0000 7f4c37d3d700 1 -- 172.21.15.73:0/4246269502 shutdown_connections 2024-04-27T03:56:42.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.916+0000 7f4c37d3d700 1 --2- 172.21.15.73:0/4246269502 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f4c1c038910 0x7f4c1c03add0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:42.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.916+0000 7f4c37d3d700 1 --2- 172.21.15.73:0/4246269502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c30106800 0x7f4c30196090 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:42.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.916+0000 7f4c37d3d700 1 -- 172.21.15.73:0/4246269502 >> 172.21.15.73:0/4246269502 conn(0x7f4c30075290 msgr2=0x7f4c300fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:42.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.916+0000 7f4c37d3d700 1 -- 172.21.15.73:0/4246269502 shutdown_connections 2024-04-27T03:56:42.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:42.916+0000 7f4c37d3d700 1 -- 172.21.15.73:0/4246269502 wait complete. 2024-04-27T03:56:42.919 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:56:43.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:43 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/4246269502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:56:45.200 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:56:45.200 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:56:45.404 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T03:56:46.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:45 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:46.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:45 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:46.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:45 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:46.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:45 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:46.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:45 smithi046 ceph-mon[26899]: Deploying daemon node-exporter.smithi073 on smithi073 2024-04-27T03:56:46.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.865+0000 7f2c0049f700 1 -- 172.21.15.73:0/3606862992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bf8105730 msgr2=0x7f2bf8105b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:46.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.865+0000 7f2c0049f700 1 --2- 172.21.15.73:0/3606862992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bf8105730 0x7f2bf8105b10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2be8009910 tx=0x7f2be8009c20 comp rx=0 tx=0).stop 2024-04-27T03:56:46.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.866+0000 7f2c0049f700 1 -- 172.21.15.73:0/3606862992 shutdown_connections 2024-04-27T03:56:46.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.866+0000 7f2c0049f700 1 --2- 172.21.15.73:0/3606862992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bf8105730 0x7f2bf8105b10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:46.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.866+0000 7f2c0049f700 1 -- 172.21.15.73:0/3606862992 >> 172.21.15.73:0/3606862992 conn(0x7f2bf80fb220 msgr2=0x7f2bf80fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:46.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.866+0000 7f2c0049f700 1 -- 172.21.15.73:0/3606862992 shutdown_connections 2024-04-27T03:56:46.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.866+0000 7f2c0049f700 1 -- 172.21.15.73:0/3606862992 wait complete. 2024-04-27T03:56:46.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.866+0000 7f2c0049f700 1 Processor -- start 2024-04-27T03:56:46.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.866+0000 7f2c0049f700 1 -- start start 2024-04-27T03:56:46.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.866+0000 7f2c0049f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bf8105730 0x7f2bf81960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:46.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.866+0000 7f2c0049f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf8196610 con 0x7f2bf8105730 2024-04-27T03:56:46.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.867+0000 7f2bfe23b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bf8105730 0x7f2bf81960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:46.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.867+0000 7f2bfe23b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bf8105730 0x7f2bf81960d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:34418/0 (socket says 172.21.15.73:34418) 2024-04-27T03:56:46.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.867+0000 7f2bfe23b700 1 -- 172.21.15.73:0/2267182871 learned_addr learned my addr 172.21.15.73:0/2267182871 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:56:46.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.867+0000 7f2bfe23b700 1 -- 172.21.15.73:0/2267182871 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be80095f0 con 0x7f2bf8105730 2024-04-27T03:56:46.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.867+0000 7f2bfe23b700 1 --2- 172.21.15.73:0/2267182871 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bf8105730 0x7f2bf81960d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2be8005490 tx=0x7f2be8005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:46.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.867+0000 7f2bef7fe700 1 -- 172.21.15.73:0/2267182871 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2be801d070 con 0x7f2bf8105730 2024-04-27T03:56:46.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.867+0000 7f2c0049f700 1 -- 172.21.15.73:0/2267182871 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf8196870 con 0x7f2bf8105730 2024-04-27T03:56:46.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.867+0000 7f2bef7fe700 1 -- 172.21.15.73:0/2267182871 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2be8005700 con 0x7f2bf8105730 2024-04-27T03:56:46.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.867+0000 7f2bef7fe700 1 -- 172.21.15.73:0/2267182871 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2be80038e0 con 0x7f2bf8105730 2024-04-27T03:56:46.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.867+0000 7f2c0049f700 1 -- 172.21.15.73:0/2267182871 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf8196cf0 con 0x7f2bf8105730 2024-04-27T03:56:46.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.868+0000 7f2c0049f700 1 -- 172.21.15.73:0/2267182871 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf8100e70 con 0x7f2bf8105730 2024-04-27T03:56:46.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.868+0000 7f2bef7fe700 1 -- 172.21.15.73:0/2267182871 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7f2be8003ba0 con 0x7f2bf8105730 2024-04-27T03:56:46.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.869+0000 7f2bef7fe700 1 --2- 172.21.15.73:0/2267182871 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2be40388c0 0x7f2be403ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:46.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.869+0000 7f2bef7fe700 1 -- 172.21.15.73:0/2267182871 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f2be804c680 con 0x7f2bf8105730 2024-04-27T03:56:46.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.869+0000 7f2bfda3a700 1 --2- 172.21.15.73:0/2267182871 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2be40388c0 0x7f2be403ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:46.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.869+0000 7f2bfda3a700 1 --2- 172.21.15.73:0/2267182871 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2be40388c0 0x7f2be403ad80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4006fd0 tx=0x7f2bf4006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:46.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:46.874+0000 7f2bef7fe700 1 -- 172.21.15.73:0/2267182871 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f2be802a590 con 0x7f2bf8105730 2024-04-27T03:56:47.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:47.150+0000 7f2c0049f700 1 -- 172.21.15.73:0/2267182871 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bf8109b20 con 0x7f2bf8105730 2024-04-27T03:56:47.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:47.150+0000 7f2bef7fe700 1 -- 172.21.15.73:0/2267182871 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f2be8026030 con 0x7f2bf8105730 2024-04-27T03:56:47.153 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:56:47.153 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:56:47.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:47.152+0000 7f2c0049f700 1 -- 172.21.15.73:0/2267182871 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2be40388c0 msgr2=0x7f2be403ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:47.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:47.152+0000 7f2c0049f700 1 --2- 172.21.15.73:0/2267182871 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2be40388c0 0x7f2be403ad80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4006fd0 tx=0x7f2bf4006e40 comp rx=0 tx=0).stop 2024-04-27T03:56:47.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:47.152+0000 7f2c0049f700 1 -- 172.21.15.73:0/2267182871 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bf8105730 msgr2=0x7f2bf81960d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:47.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:47.152+0000 7f2c0049f700 1 --2- 172.21.15.73:0/2267182871 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bf8105730 0x7f2bf81960d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2be8005490 tx=0x7f2be8005570 comp rx=0 tx=0).stop 2024-04-27T03:56:47.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:47.152+0000 7f2c0049f700 1 -- 172.21.15.73:0/2267182871 shutdown_connections 2024-04-27T03:56:47.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:47.152+0000 7f2c0049f700 1 --2- 172.21.15.73:0/2267182871 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2be40388c0 0x7f2be403ad80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:47.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:47.152+0000 7f2c0049f700 1 --2- 172.21.15.73:0/2267182871 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bf8105730 0x7f2bf81960d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:47.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:47.152+0000 7f2c0049f700 1 -- 172.21.15.73:0/2267182871 >> 172.21.15.73:0/2267182871 conn(0x7f2bf80fb220 msgr2=0x7f2bf8068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:47.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:47.153+0000 7f2c0049f700 1 -- 172.21.15.73:0/2267182871 shutdown_connections 2024-04-27T03:56:47.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:47.153+0000 7f2c0049f700 1 -- 172.21.15.73:0/2267182871 wait complete. 2024-04-27T03:56:47.156 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:56:47.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:46 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:47.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:46 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:56:48.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:47 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/2267182871' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:56:49.272 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:56:49.272 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:56:49.709 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T03:56:51.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:51.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:51.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:51.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:51.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi073.awyllt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T03:56:51.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi073.awyllt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-27T03:56:51.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T03:56:51.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:56:51.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:50 smithi046 ceph-mon[26899]: Deploying daemon mgr.smithi073.awyllt on smithi073 2024-04-27T03:56:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.254+0000 7f996231b700 1 -- 172.21.15.73:0/135284393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c073310 msgr2=0x7f995c0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.254+0000 7f996231b700 1 --2- 172.21.15.73:0/135284393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c073310 0x7f995c0736f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f994c009910 tx=0x7f994c009c20 comp rx=0 tx=0).stop 2024-04-27T03:56:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.254+0000 7f996231b700 1 -- 172.21.15.73:0/135284393 shutdown_connections 2024-04-27T03:56:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.254+0000 7f996231b700 1 --2- 172.21.15.73:0/135284393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c073310 0x7f995c0736f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.254+0000 7f996231b700 1 -- 172.21.15.73:0/135284393 >> 172.21.15.73:0/135284393 conn(0x7f995c0fc310 msgr2=0x7f995c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.254+0000 7f996231b700 1 -- 172.21.15.73:0/135284393 shutdown_connections 2024-04-27T03:56:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.254+0000 7f996231b700 1 -- 172.21.15.73:0/135284393 wait complete. 2024-04-27T03:56:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.255+0000 7f996231b700 1 Processor -- start 2024-04-27T03:56:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.255+0000 7f996231b700 1 -- start start 2024-04-27T03:56:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.255+0000 7f996231b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c073310 0x7f995c198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.255+0000 7f996231b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995c198890 con 0x7f995c073310 2024-04-27T03:56:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.256+0000 7f995bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c073310 0x7f995c198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:51.258 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.256+0000 7f995bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c073310 0x7f995c198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:34444/0 (socket says 172.21.15.73:34444) 2024-04-27T03:56:51.258 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.256+0000 7f995bfff700 1 -- 172.21.15.73:0/3923651419 learned_addr learned my addr 172.21.15.73:0/3923651419 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:56:51.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.256+0000 7f995bfff700 1 -- 172.21.15.73:0/3923651419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0095f0 con 0x7f995c073310 2024-04-27T03:56:51.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.256+0000 7f995bfff700 1 --2- 172.21.15.73:0/3923651419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c073310 0x7f995c198350 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f994c005490 tx=0x7f994c005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:51.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.256+0000 7f99597fa700 1 -- 172.21.15.73:0/3923651419 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f994c01d070 con 0x7f995c073310 2024-04-27T03:56:51.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.256+0000 7f996231b700 1 -- 172.21.15.73:0/3923651419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995c198a90 con 0x7f995c073310 2024-04-27T03:56:51.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.256+0000 7f99597fa700 1 -- 172.21.15.73:0/3923651419 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f994c005700 con 0x7f995c073310 2024-04-27T03:56:51.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.256+0000 7f99597fa700 1 -- 172.21.15.73:0/3923651419 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f994c003950 con 0x7f995c073310 2024-04-27T03:56:51.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.256+0000 7f996231b700 1 -- 172.21.15.73:0/3923651419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f995c198eb0 con 0x7f995c073310 2024-04-27T03:56:51.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.257+0000 7f996231b700 1 -- 172.21.15.73:0/3923651419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f995c0749c0 con 0x7f995c073310 2024-04-27T03:56:51.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.257+0000 7f99597fa700 1 -- 172.21.15.73:0/3923651419 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7f994c003c10 con 0x7f995c073310 2024-04-27T03:56:51.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.257+0000 7f99597fa700 1 --2- 172.21.15.73:0/3923651419 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f99440388c0 0x7f994403ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:51.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.257+0000 7f99597fa700 1 -- 172.21.15.73:0/3923651419 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f994c04c6d0 con 0x7f995c073310 2024-04-27T03:56:51.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.258+0000 7f995b7fe700 1 --2- 172.21.15.73:0/3923651419 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f99440388c0 0x7f994403ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:51.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.258+0000 7f995b7fe700 1 --2- 172.21.15.73:0/3923651419 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f99440388c0 0x7f994403ad80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9950006fd0 tx=0x7f9950006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:51.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.263+0000 7f99597fa700 1 -- 172.21.15.73:0/3923651419 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f994c02a590 con 0x7f995c073310 2024-04-27T03:56:51.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.546+0000 7f996231b700 1 -- 172.21.15.73:0/3923651419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f995c10ce60 con 0x7f995c073310 2024-04-27T03:56:51.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.546+0000 7f99597fa700 1 -- 172.21.15.73:0/3923651419 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f994c026070 con 0x7f995c073310 2024-04-27T03:56:51.548 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:56:51.548 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:56:51.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.548+0000 7f996231b700 1 -- 172.21.15.73:0/3923651419 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f99440388c0 msgr2=0x7f994403ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:51.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.548+0000 7f996231b700 1 --2- 172.21.15.73:0/3923651419 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f99440388c0 0x7f994403ad80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9950006fd0 tx=0x7f9950006e40 comp rx=0 tx=0).stop 2024-04-27T03:56:51.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.548+0000 7f996231b700 1 -- 172.21.15.73:0/3923651419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c073310 msgr2=0x7f995c198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:51.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.548+0000 7f996231b700 1 --2- 172.21.15.73:0/3923651419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c073310 0x7f995c198350 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f994c005490 tx=0x7f994c005570 comp rx=0 tx=0).stop 2024-04-27T03:56:51.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.549+0000 7f996231b700 1 -- 172.21.15.73:0/3923651419 shutdown_connections 2024-04-27T03:56:51.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.549+0000 7f996231b700 1 --2- 172.21.15.73:0/3923651419 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f99440388c0 0x7f994403ad80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:51.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.549+0000 7f996231b700 1 --2- 172.21.15.73:0/3923651419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c073310 0x7f995c198350 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:51.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.549+0000 7f996231b700 1 -- 172.21.15.73:0/3923651419 >> 172.21.15.73:0/3923651419 conn(0x7f995c0fc310 msgr2=0x7f995c104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:51.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.549+0000 7f996231b700 1 -- 172.21.15.73:0/3923651419 shutdown_connections 2024-04-27T03:56:51.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:51.549+0000 7f996231b700 1 -- 172.21.15.73:0/3923651419 wait complete. 2024-04-27T03:56:51.551 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:56:51.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:51 smithi046 ceph-mon[26899]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:56:51.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:51 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:51.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:51 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/3923651419' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:56:53.969 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:56:53.969 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:56:54.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:54 smithi046 ceph-mon[26899]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:56:54.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:54 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:54.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:54 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:54.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:54 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:54.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:54 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:54.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:54 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T03:56:54.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:54 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:56:54.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:54 smithi046 ceph-mon[26899]: Deploying daemon mon.smithi073 on smithi073 2024-04-27T03:56:56.064 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi073/config 2024-04-27T03:56:56.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:56 smithi046 ceph-mon[26899]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:56:56.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:56:56 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:56:57.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.717+0000 7f40e1bda700 1 -- 172.21.15.73:0/3044701480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc101af0 msgr2=0x7f40dc101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:57.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.717+0000 7f40e1bda700 1 --2- 172.21.15.73:0/3044701480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc101af0 0x7f40dc101ed0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f40cc009910 tx=0x7f40cc009c20 comp rx=0 tx=0).stop 2024-04-27T03:56:57.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.717+0000 7f40e1bda700 1 -- 172.21.15.73:0/3044701480 shutdown_connections 2024-04-27T03:56:57.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.717+0000 7f40e1bda700 1 --2- 172.21.15.73:0/3044701480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc101af0 0x7f40dc101ed0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:57.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.717+0000 7f40e1bda700 1 -- 172.21.15.73:0/3044701480 >> 172.21.15.73:0/3044701480 conn(0x7f40dc0fb340 msgr2=0x7f40dc0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:57.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.717+0000 7f40e1bda700 1 -- 172.21.15.73:0/3044701480 shutdown_connections 2024-04-27T03:56:57.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.717+0000 7f40e1bda700 1 -- 172.21.15.73:0/3044701480 wait complete. 2024-04-27T03:56:57.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.718+0000 7f40e1bda700 1 Processor -- start 2024-04-27T03:56:57.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.718+0000 7f40e1bda700 1 -- start start 2024-04-27T03:56:57.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.718+0000 7f40e1bda700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc101af0 0x7f40dc198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:57.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.718+0000 7f40e1bda700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc1989a0 con 0x7f40dc101af0 2024-04-27T03:56:57.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.718+0000 7f40db7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc101af0 0x7f40dc198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:57.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.719+0000 7f40db7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc101af0 0x7f40dc198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.73:40798/0 (socket says 172.21.15.73:40798) 2024-04-27T03:56:57.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.719+0000 7f40db7fe700 1 -- 172.21.15.73:0/3208034052 learned_addr learned my addr 172.21.15.73:0/3208034052 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:56:57.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.719+0000 7f40db7fe700 1 -- 172.21.15.73:0/3208034052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc0095f0 con 0x7f40dc101af0 2024-04-27T03:56:57.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.719+0000 7f40db7fe700 1 --2- 172.21.15.73:0/3208034052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc101af0 0x7f40dc198460 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f40cc005490 tx=0x7f40cc005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:57.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.719+0000 7f40d8ff9700 1 -- 172.21.15.73:0/3208034052 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40cc01d070 con 0x7f40dc101af0 2024-04-27T03:56:57.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.719+0000 7f40d8ff9700 1 -- 172.21.15.73:0/3208034052 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f40cc005700 con 0x7f40dc101af0 2024-04-27T03:56:57.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.719+0000 7f40d8ff9700 1 -- 172.21.15.73:0/3208034052 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40cc003950 con 0x7f40dc101af0 2024-04-27T03:56:57.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.719+0000 7f40e1bda700 1 -- 172.21.15.73:0/3208034052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40dc198ba0 con 0x7f40dc101af0 2024-04-27T03:56:57.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.719+0000 7f40e1bda700 1 -- 172.21.15.73:0/3208034052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40dc198fc0 con 0x7f40dc101af0 2024-04-27T03:56:57.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.720+0000 7f40d8ff9700 1 -- 172.21.15.73:0/3208034052 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7f40cc003bc0 con 0x7f40dc101af0 2024-04-27T03:56:57.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.721+0000 7f40d8ff9700 1 --2- 172.21.15.73:0/3208034052 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f40c4038910 0x7f40c403add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:56:57.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.721+0000 7f40d8ff9700 1 -- 172.21.15.73:0/3208034052 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f40cc0507e0 con 0x7f40dc101af0 2024-04-27T03:56:57.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.721+0000 7f40e1bda700 1 -- 172.21.15.73:0/3208034052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40dc0696a0 con 0x7f40dc101af0 2024-04-27T03:56:57.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.721+0000 7f40daffd700 1 --2- 172.21.15.73:0/3208034052 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f40c4038910 0x7f40c403add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:56:57.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.721+0000 7f40daffd700 1 --2- 172.21.15.73:0/3208034052 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f40c4038910 0x7f40c403add0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f40d0006fd0 tx=0x7f40d0006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:56:57.728 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:57.726+0000 7f40d8ff9700 1 -- 172.21.15.73:0/3208034052 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f40cc026070 con 0x7f40dc101af0 2024-04-27T03:56:58.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:58.024+0000 7f40e1bda700 1 -- 172.21.15.73:0/3208034052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40dc062510 con 0x7f40dc101af0 2024-04-27T03:56:58.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:58.025+0000 7f40d8ff9700 1 -- 172.21.15.73:0/3208034052 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f40cc015940 con 0x7f40dc101af0 2024-04-27T03:56:58.027 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:56:58.028 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:54:05.475265Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T03:56:58.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:58.027+0000 7f40e1bda700 1 -- 172.21.15.73:0/3208034052 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f40c4038910 msgr2=0x7f40c403add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:58.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:58.027+0000 7f40e1bda700 1 --2- 172.21.15.73:0/3208034052 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f40c4038910 0x7f40c403add0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f40d0006fd0 tx=0x7f40d0006e40 comp rx=0 tx=0).stop 2024-04-27T03:56:58.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:58.027+0000 7f40e1bda700 1 -- 172.21.15.73:0/3208034052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc101af0 msgr2=0x7f40dc198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:56:58.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:58.027+0000 7f40e1bda700 1 --2- 172.21.15.73:0/3208034052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc101af0 0x7f40dc198460 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f40cc005490 tx=0x7f40cc005570 comp rx=0 tx=0).stop 2024-04-27T03:56:58.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:58.027+0000 7f40e1bda700 1 -- 172.21.15.73:0/3208034052 shutdown_connections 2024-04-27T03:56:58.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:58.027+0000 7f40e1bda700 1 --2- 172.21.15.73:0/3208034052 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f40c4038910 0x7f40c403add0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:58.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:58.027+0000 7f40e1bda700 1 --2- 172.21.15.73:0/3208034052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc101af0 0x7f40dc198460 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:56:58.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:58.027+0000 7f40e1bda700 1 -- 172.21.15.73:0/3208034052 >> 172.21.15.73:0/3208034052 conn(0x7f40dc0fb340 msgr2=0x7f40dc106cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:56:58.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:58.027+0000 7f40e1bda700 1 -- 172.21.15.73:0/3208034052 shutdown_connections 2024-04-27T03:56:58.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:56:58.027+0000 7f40e1bda700 1 -- 172.21.15.73:0/3208034052 wait complete. 2024-04-27T03:56:58.030 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2024-04-27T03:56:59.747 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T03:56:59.747 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T03:56:59.950 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi073/config 2024-04-27T03:57:03.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-04-27T03:57:03.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: mon.smithi046 calling monitor election 2024-04-27T03:57:03.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-04-27T03:57:03.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:03.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-04-27T03:57:03.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-04-27T03:57:03.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: mon.smithi073 calling monitor election 2024-04-27T03:57:03.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.? 172.21.15.73:0/3087852792' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi073.awyllt/crt"}]: dispatch 2024-04-27T03:57:03.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:03.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:57:03.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-04-27T03:57:03.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-04-27T03:57:03.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-04-27T03:57:03.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: mon.smithi046 is new leader, mons smithi046,smithi073 in quorum (ranks 0,1) 2024-04-27T03:57:03.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: monmap e2: 2 mons at {smithi046=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],smithi073=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-27T03:57:03.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: fsmap 2024-04-27T03:57:03.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: osdmap e5: 0 total, 0 up, 0 in 2024-04-27T03:57:03.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: mgrmap e17: smithi046.qrfdvr(active, since 32s) 2024-04-27T03:57:03.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: overall HEALTH_OK 2024-04-27T03:57:03.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: Standby manager daemon smithi073.awyllt started 2024-04-27T03:57:03.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.? 172.21.15.73:0/3087852792' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T03:57:03.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:03.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.? 172.21.15.73:0/3087852792' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi073.awyllt/key"}]: dispatch 2024-04-27T03:57:03.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.? 172.21.15.73:0/3087852792' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T03:57:03.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:03.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:03.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:03.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:03.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:03 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:04.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.566+0000 7f8f4d7dd700 1 -- 172.21.15.73:0/2696995466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f340035c0 msgr2=0x7f8f34005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:57:04.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.566+0000 7f8f4d7dd700 1 --2- 172.21.15.73:0/2696995466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f340035c0 0x7f8f34005a50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8f480ff750 tx=0x7f8f3800feb0 comp rx=0 tx=0).stop 2024-04-27T03:57:04.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.566+0000 7f8f4d7dd700 1 -- 172.21.15.73:0/2696995466 shutdown_connections 2024-04-27T03:57:04.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.566+0000 7f8f4d7dd700 1 --2- 172.21.15.73:0/2696995466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f340035c0 0x7f8f34005a50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:04.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.566+0000 7f8f4d7dd700 1 -- 172.21.15.73:0/2696995466 >> 172.21.15.73:0/2696995466 conn(0x7f8f480fb390 msgr2=0x7f8f480fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:57:04.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.566+0000 7f8f4d7dd700 1 -- 172.21.15.73:0/2696995466 shutdown_connections 2024-04-27T03:57:04.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.566+0000 7f8f4d7dd700 1 -- 172.21.15.73:0/2696995466 wait complete. 2024-04-27T03:57:04.569 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:04 smithi073 ceph-mon[34160]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:04.570 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:04 smithi073 ceph-mon[34160]: mgrmap e18: smithi046.qrfdvr(active, since 32s), standbys: smithi073.awyllt 2024-04-27T03:57:04.570 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:04 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mgr metadata", "who": "smithi073.awyllt", "id": "smithi073.awyllt"}]: dispatch 2024-04-27T03:57:04.570 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:04 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:57:04.570 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:04 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-04-27T03:57:04.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.566+0000 7f8f4d7dd700 1 Processor -- start 2024-04-27T03:57:04.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.566+0000 7f8f4d7dd700 1 -- start start 2024-04-27T03:57:04.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.567+0000 7f8f4d7dd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f340035c0 0x7f8f48197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:57:04.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.567+0000 7f8f4d7dd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f48197660 0x7f8f48197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:57:04.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.567+0000 7f8f4d7dd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f481a0190 con 0x7f8f48197660 2024-04-27T03:57:04.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.567+0000 7f8f4d7dd700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f481a02d0 con 0x7f8f340035c0 2024-04-27T03:57:04.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.567+0000 7f8f46ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f340035c0 0x7f8f48197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:57:04.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.567+0000 7f8f46ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f340035c0 0x7f8f48197120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41926/0 (socket says 172.21.15.73:41926) 2024-04-27T03:57:04.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.567+0000 7f8f46ffd700 1 -- 172.21.15.73:0/1099172006 learned_addr learned my addr 172.21.15.73:0/1099172006 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:57:04.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.567+0000 7f8f46ffd700 1 -- 172.21.15.73:0/1099172006 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f340035c0 msgr2=0x7f8f48197120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T03:57:04.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.567+0000 7f8f46ffd700 1 -- 172.21.15.73:0/1099172006 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f340035c0 msgr2=0x7f8f48197120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T03:57:04.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.567+0000 7f8f46ffd700 1 --2- 172.21.15.73:0/1099172006 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f340035c0 0x7f8f48197120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T03:57:04.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.567+0000 7f8f46ffd700 1 --2- 172.21.15.73:0/1099172006 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f340035c0 0x7f8f48197120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-27T03:57:04.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.567+0000 7f8f3ffff700 1 --2- 172.21.15.73:0/1099172006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f48197660 0x7f8f48197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:57:04.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.567+0000 7f8f3ffff700 1 -- 172.21.15.73:0/1099172006 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f340035c0 msgr2=0x7f8f48197120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T03:57:04.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.567+0000 7f8f3ffff700 1 --2- 172.21.15.73:0/1099172006 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f340035c0 0x7f8f48197120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:04.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.567+0000 7f8f3ffff700 1 -- 172.21.15.73:0/1099172006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3800f900 con 0x7f8f48197660 2024-04-27T03:57:04.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.568+0000 7f8f3ffff700 1 --2- 172.21.15.73:0/1099172006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f48197660 0x7f8f48197ae0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8f3000b500 tx=0x7f8f3000b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:57:04.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.568+0000 7f8f44ff9700 1 -- 172.21.15.73:0/1099172006 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f30011800 con 0x7f8f48197660 2024-04-27T03:57:04.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.568+0000 7f8f44ff9700 1 -- 172.21.15.73:0/1099172006 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f30011e20 con 0x7f8f48197660 2024-04-27T03:57:04.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.568+0000 7f8f4d7dd700 1 -- 172.21.15.73:0/1099172006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f481a0470 con 0x7f8f48197660 2024-04-27T03:57:04.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.568+0000 7f8f44ff9700 1 -- 172.21.15.73:0/1099172006 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f3000f2e0 con 0x7f8f48197660 2024-04-27T03:57:04.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.568+0000 7f8f4d7dd700 1 -- 172.21.15.73:0/1099172006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f481a0990 con 0x7f8f48197660 2024-04-27T03:57:04.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.569+0000 7f8f4d7dd700 1 -- 172.21.15.73:0/1099172006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f4818f080 con 0x7f8f48197660 2024-04-27T03:57:04.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.569+0000 7f8f44ff9700 1 -- 172.21.15.73:0/1099172006 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7f8f30010380 con 0x7f8f48197660 2024-04-27T03:57:04.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.570+0000 7f8f44ff9700 1 --2- 172.21.15.73:0/1099172006 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8f2806cd60 0x7f8f2806f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:57:04.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.570+0000 7f8f44ff9700 1 -- 172.21.15.73:0/1099172006 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f8f3008ab40 con 0x7f8f48197660 2024-04-27T03:57:04.575 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.570+0000 7f8f46ffd700 1 --2- 172.21.15.73:0/1099172006 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8f2806cd60 0x7f8f2806f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:57:04.575 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.570+0000 7f8f46ffd700 1 --2- 172.21.15.73:0/1099172006 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8f2806cd60 0x7f8f2806f220 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8f380059d0 tx=0x7f8f38005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:57:04.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.574+0000 7f8f44ff9700 1 -- 172.21.15.73:0/1099172006 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f8f30055fd0 con 0x7f8f48197660 2024-04-27T03:57:04.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:04 smithi046 ceph-mon[26899]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:04.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:04 smithi046 ceph-mon[26899]: mgrmap e18: smithi046.qrfdvr(active, since 32s), standbys: smithi073.awyllt 2024-04-27T03:57:04.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:04 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mgr metadata", "who": "smithi073.awyllt", "id": "smithi073.awyllt"}]: dispatch 2024-04-27T03:57:04.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:04 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:57:04.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:04 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-04-27T03:57:04.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.860+0000 7f8f4d7dd700 1 -- 172.21.15.73:0/1099172006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f48100f60 con 0x7f8f48197660 2024-04-27T03:57:04.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.861+0000 7f8f44ff9700 1 -- 172.21.15.73:0/1099172006 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1018 (secure 0 0 0) 0x7f8f30015410 con 0x7f8f48197660 2024-04-27T03:57:04.864 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-04-27T03:57:04.864 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":2,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","modified":"2024-04-27T03:56:58.145455Z","created":"2024-04-27T03:54:05.475265Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-27T03:57:04.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.863+0000 7f8f4d7dd700 1 -- 172.21.15.73:0/1099172006 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8f2806cd60 msgr2=0x7f8f2806f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:57:04.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.863+0000 7f8f4d7dd700 1 --2- 172.21.15.73:0/1099172006 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8f2806cd60 0x7f8f2806f220 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8f380059d0 tx=0x7f8f38005960 comp rx=0 tx=0).stop 2024-04-27T03:57:04.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.863+0000 7f8f4d7dd700 1 -- 172.21.15.73:0/1099172006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f48197660 msgr2=0x7f8f48197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:57:04.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.863+0000 7f8f4d7dd700 1 --2- 172.21.15.73:0/1099172006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f48197660 0x7f8f48197ae0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8f3000b500 tx=0x7f8f3000b8c0 comp rx=0 tx=0).stop 2024-04-27T03:57:04.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.863+0000 7f8f4d7dd700 1 -- 172.21.15.73:0/1099172006 shutdown_connections 2024-04-27T03:57:04.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.863+0000 7f8f4d7dd700 1 --2- 172.21.15.73:0/1099172006 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8f2806cd60 0x7f8f2806f220 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:04.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.863+0000 7f8f4d7dd700 1 --2- 172.21.15.73:0/1099172006 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f340035c0 0x7f8f48197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:04.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.863+0000 7f8f4d7dd700 1 --2- 172.21.15.73:0/1099172006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f48197660 0x7f8f48197ae0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:04.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.863+0000 7f8f4d7dd700 1 -- 172.21.15.73:0/1099172006 >> 172.21.15.73:0/1099172006 conn(0x7f8f480fb390 msgr2=0x7f8f480fd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:57:04.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.863+0000 7f8f4d7dd700 1 -- 172.21.15.73:0/1099172006 shutdown_connections 2024-04-27T03:57:04.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:57:04.863+0000 7f8f4d7dd700 1 -- 172.21.15.73:0/1099172006 wait complete. 2024-04-27T03:57:04.867 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 2 2024-04-27T03:57:05.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:05 smithi073 ceph-mon[34160]: from='client.? 172.21.15.73:0/1099172006' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:57:05.639 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-27T03:57:05.639 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-27T03:57:05.667 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:05 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/1099172006' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T03:57:05.867 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T03:57:06.305 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:06 smithi046 ceph-mon[26899]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:06.306 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:06 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:06.306 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:06 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:06.306 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:06 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:06.306 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:06 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:06.306 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:06 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:57:06.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:06 smithi073 ceph-mon[34160]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:06.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:06 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:06.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:06 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:06.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:06 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:06.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:06 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:06.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:06 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:57:07.344 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:07 smithi046 ceph-mon[26899]: Updating smithi046:/etc/ceph/ceph.conf 2024-04-27T03:57:07.344 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:07 smithi046 ceph-mon[26899]: Updating smithi073:/etc/ceph/ceph.conf 2024-04-27T03:57:07.344 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:07 smithi046 ceph-mon[26899]: Updating smithi073:/var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T03:57:07.344 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:07 smithi046 ceph-mon[26899]: Updating smithi046:/var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T03:57:07.344 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:07 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:07.344 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:07 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:07.344 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:07 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:07.344 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:07 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:07.344 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:07 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:07.344 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:07 smithi046 ceph-mon[26899]: Reconfiguring mon.smithi046 (unknown last config time)... 2024-04-27T03:57:07.345 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:07 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T03:57:07.345 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:07 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T03:57:07.345 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:07 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:07.345 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:07 smithi046 ceph-mon[26899]: Reconfiguring daemon mon.smithi046 on smithi046 2024-04-27T03:57:07.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:07 smithi073 ceph-mon[34160]: Updating smithi046:/etc/ceph/ceph.conf 2024-04-27T03:57:07.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:07 smithi073 ceph-mon[34160]: Updating smithi073:/etc/ceph/ceph.conf 2024-04-27T03:57:07.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:07 smithi073 ceph-mon[34160]: Updating smithi073:/var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T03:57:07.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:07 smithi073 ceph-mon[34160]: Updating smithi046:/var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T03:57:07.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:07 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:07.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:07 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:07.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:07 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:07.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:07 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:07.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:07 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:07.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:07 smithi073 ceph-mon[34160]: Reconfiguring mon.smithi046 (unknown last config time)... 2024-04-27T03:57:07.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:07 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T03:57:07.578 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:07 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T03:57:07.578 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:07 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:07.578 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:07 smithi073 ceph-mon[34160]: Reconfiguring daemon mon.smithi046 on smithi046 2024-04-27T03:57:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.890+0000 7fc260bd9700 1 -- 172.21.15.46:0/3397636868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc25c101a50 msgr2=0x7fc25c101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:57:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.890+0000 7fc260bd9700 1 --2- 172.21.15.46:0/3397636868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc25c101a50 0x7fc25c101e30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009910 tx=0x7fc24c009c20 comp rx=0 tx=0).stop 2024-04-27T03:57:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.890+0000 7fc260bd9700 1 -- 172.21.15.46:0/3397636868 shutdown_connections 2024-04-27T03:57:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.890+0000 7fc260bd9700 1 --2- 172.21.15.46:0/3397636868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc25c101a50 0x7fc25c101e30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.890+0000 7fc260bd9700 1 -- 172.21.15.46:0/3397636868 >> 172.21.15.46:0/3397636868 conn(0x7fc25c0fb220 msgr2=0x7fc25c0fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:57:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.890+0000 7fc260bd9700 1 -- 172.21.15.46:0/3397636868 shutdown_connections 2024-04-27T03:57:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.890+0000 7fc260bd9700 1 -- 172.21.15.46:0/3397636868 wait complete. 2024-04-27T03:57:08.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.891+0000 7fc260bd9700 1 Processor -- start 2024-04-27T03:57:08.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.891+0000 7fc260bd9700 1 -- start start 2024-04-27T03:57:08.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.891+0000 7fc260bd9700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc25c101a50 0x7fc25c0ff6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:57:08.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.891+0000 7fc260bd9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc25c0ffbf0 0x7fc25c100070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:57:08.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.891+0000 7fc260bd9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c101600 con 0x7fc25c0ffbf0 2024-04-27T03:57:08.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.891+0000 7fc260bd9700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c1a2560 con 0x7fc25c101a50 2024-04-27T03:57:08.301 INFO:teuthology.orchestra.run.smithi046.stdout:# minimal ceph.conf for b1d3a2d0-0449-11ef-bc93-c7b262605968 2024-04-27T03:57:08.301 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2024-04-27T03:57:08.301 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = b1d3a2d0-0449-11ef-bc93-c7b262605968 2024-04-27T03:57:08.302 INFO:teuthology.orchestra.run.smithi046.stdout: mon_host = [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] 2024-04-27T03:57:08.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.891+0000 7fc259d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc25c0ffbf0 0x7fc25c100070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:57:08.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.891+0000 7fc259d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc25c0ffbf0 0x7fc25c100070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50188/0 (socket says 172.21.15.46:50188) 2024-04-27T03:57:08.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.891+0000 7fc259d9b700 1 -- 172.21.15.46:0/3082398677 learned_addr learned my addr 172.21.15.46:0/3082398677 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:57:08.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.892+0000 7fc25a59c700 1 --2- 172.21.15.46:0/3082398677 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc25c101a50 0x7fc25c0ff6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:57:08.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.892+0000 7fc259d9b700 1 -- 172.21.15.46:0/3082398677 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc25c101a50 msgr2=0x7fc25c0ff6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:57:08.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.892+0000 7fc259d9b700 1 --2- 172.21.15.46:0/3082398677 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc25c101a50 0x7fc25c0ff6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:08.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.892+0000 7fc259d9b700 1 -- 172.21.15.46:0/3082398677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24c0095f0 con 0x7fc25c0ffbf0 2024-04-27T03:57:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.892+0000 7fc259d9b700 1 --2- 172.21.15.46:0/3082398677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc25c0ffbf0 0x7fc25c100070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc25000b700 tx=0x7fc25000bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:57:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.892+0000 7fc24b7fe700 1 -- 172.21.15.46:0/3082398677 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc250011800 con 0x7fc25c0ffbf0 2024-04-27T03:57:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.892+0000 7fc260bd9700 1 -- 172.21.15.46:0/3082398677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc25c1a2760 con 0x7fc25c0ffbf0 2024-04-27T03:57:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.892+0000 7fc24b7fe700 1 -- 172.21.15.46:0/3082398677 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc250011e20 con 0x7fc25c0ffbf0 2024-04-27T03:57:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.892+0000 7fc24b7fe700 1 -- 172.21.15.46:0/3082398677 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc25000f2e0 con 0x7fc25c0ffbf0 2024-04-27T03:57:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.892+0000 7fc260bd9700 1 -- 172.21.15.46:0/3082398677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc25c1a2c80 con 0x7fc25c0ffbf0 2024-04-27T03:57:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.893+0000 7fc260bd9700 1 -- 172.21.15.46:0/3082398677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc25c069120 con 0x7fc25c0ffbf0 2024-04-27T03:57:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.894+0000 7fc24b7fe700 1 -- 172.21.15.46:0/3082398677 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7fc25000f5a0 con 0x7fc25c0ffbf0 2024-04-27T03:57:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.894+0000 7fc24b7fe700 1 --2- 172.21.15.46:0/3082398677 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fc24406cdb0 0x7fc24406f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:57:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.894+0000 7fc24b7fe700 1 -- 172.21.15.46:0/3082398677 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fc25008ad40 con 0x7fc25c0ffbf0 2024-04-27T03:57:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.894+0000 7fc25a59c700 1 --2- 172.21.15.46:0/3082398677 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fc24406cdb0 0x7fc24406f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:57:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.895+0000 7fc25a59c700 1 --2- 172.21.15.46:0/3082398677 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fc24406cdb0 0x7fc24406f270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc24c006010 tx=0x7fc24c00b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:57:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:07.899+0000 7fc24b7fe700 1 -- 172.21.15.46:0/3082398677 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fc2500561f0 con 0x7fc25c0ffbf0 2024-04-27T03:57:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:08.106+0000 7fc260bd9700 1 -- 172.21.15.46:0/3082398677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fc25c062510 con 0x7fc25c0ffbf0 2024-04-27T03:57:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:08.106+0000 7fc24b7fe700 1 -- 172.21.15.46:0/3082398677 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+223 (secure 0 0 0) 0x7fc25008a500 con 0x7fc25c0ffbf0 2024-04-27T03:57:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:08.108+0000 7fc260bd9700 1 -- 172.21.15.46:0/3082398677 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fc24406cdb0 msgr2=0x7fc24406f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:57:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:08.108+0000 7fc260bd9700 1 --2- 172.21.15.46:0/3082398677 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fc24406cdb0 0x7fc24406f270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc24c006010 tx=0x7fc24c00b540 comp rx=0 tx=0).stop 2024-04-27T03:57:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:08.108+0000 7fc260bd9700 1 -- 172.21.15.46:0/3082398677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc25c0ffbf0 msgr2=0x7fc25c100070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:57:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:08.108+0000 7fc260bd9700 1 --2- 172.21.15.46:0/3082398677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc25c0ffbf0 0x7fc25c100070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc25000b700 tx=0x7fc25000bac0 comp rx=0 tx=0).stop 2024-04-27T03:57:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:08.108+0000 7fc260bd9700 1 -- 172.21.15.46:0/3082398677 shutdown_connections 2024-04-27T03:57:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:08.108+0000 7fc260bd9700 1 --2- 172.21.15.46:0/3082398677 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fc24406cdb0 0x7fc24406f270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:08.108+0000 7fc260bd9700 1 --2- 172.21.15.46:0/3082398677 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc25c101a50 0x7fc25c0ff6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:08.108+0000 7fc260bd9700 1 --2- 172.21.15.46:0/3082398677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc25c0ffbf0 0x7fc25c100070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:08.108+0000 7fc260bd9700 1 -- 172.21.15.46:0/3082398677 >> 172.21.15.46:0/3082398677 conn(0x7fc25c0fb220 msgr2=0x7fc25c104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:57:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:08.108+0000 7fc260bd9700 1 -- 172.21.15.46:0/3082398677 shutdown_connections 2024-04-27T03:57:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:08.109+0000 7fc260bd9700 1 -- 172.21.15.46:0/3082398677 wait complete. 2024-04-27T03:57:08.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:08 smithi046 ceph-mon[26899]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:08.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:08 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/3082398677' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:08.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:08 smithi073 ceph-mon[34160]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:08.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:08 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/3082398677' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:08.847 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-27T03:57:08.847 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T03:57:08.847 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.conf 2024-04-27T03:57:08.915 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T03:57:08.915 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-27T03:57:08.980 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-04-27T03:57:08.980 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/ceph.conf 2024-04-27T03:57:09.010 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-04-27T03:57:09.010 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-27T03:57:09.077 INFO:tasks.cephadm:Deploying OSDs... 2024-04-27T03:57:09.077 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T03:57:09.077 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2024-04-27T03:57:09.093 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-27T03:57:09.093 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2024-04-27T03:57:09.149 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-27T03:57:09.150 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T03:57:09.150 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-27T03:57:09.150 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T03:57:09.150 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T03:57:09.150 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-04-27 03:56:20.355401138 +0000 2024-04-27T03:57:09.150 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-27 03:50:34.956728842 +0000 2024-04-27T03:57:09.150 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-27 03:50:34.956728842 +0000 2024-04-27T03:57:09.150 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-27 03:50:34.956728842 +0000 2024-04-27T03:57:09.150 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-27T03:57:09.213 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-04-27T03:57:09.213 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-04-27T03:57:09.214 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000105619 s, 4.8 MB/s 2024-04-27T03:57:09.215 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-27T03:57:09.270 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2024-04-27T03:57:09.327 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-27T03:57:09.327 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T03:57:09.327 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-27T03:57:09.327 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T03:57:09.327 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T03:57:09.327 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-04-27 03:56:20.356401128 +0000 2024-04-27T03:57:09.327 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-27 03:50:35.301724900 +0000 2024-04-27T03:57:09.327 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-27 03:50:35.301724900 +0000 2024-04-27T03:57:09.328 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-27 03:50:35.301724900 +0000 2024-04-27T03:57:09.328 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-27T03:57:09.392 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-04-27T03:57:09.393 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-04-27T03:57:09.393 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000102508 s, 5.0 MB/s 2024-04-27T03:57:09.394 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-27T03:57:09.450 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2024-04-27T03:57:09.506 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-27T03:57:09.506 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T03:57:09.506 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-27T03:57:09.506 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T03:57:09.506 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T03:57:09.506 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-04-27 03:56:20.356401128 +0000 2024-04-27T03:57:09.506 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-27 03:50:35.636721072 +0000 2024-04-27T03:57:09.507 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-27 03:50:35.636721072 +0000 2024-04-27T03:57:09.507 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-27 03:50:35.636721072 +0000 2024-04-27T03:57:09.507 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-27T03:57:09.570 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-04-27T03:57:09.570 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-04-27T03:57:09.571 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000103133 s, 5.0 MB/s 2024-04-27T03:57:09.572 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-27T03:57:09.628 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2024-04-27T03:57:09.684 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-27T03:57:09.684 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T03:57:09.685 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-27T03:57:09.685 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T03:57:09.685 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T03:57:09.685 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-04-27 03:56:20.357401117 +0000 2024-04-27T03:57:09.685 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-27 03:50:35.975717198 +0000 2024-04-27T03:57:09.685 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-27 03:50:35.975717198 +0000 2024-04-27T03:57:09.685 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-27 03:50:35.975717198 +0000 2024-04-27T03:57:09.685 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-27T03:57:09.748 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-04-27T03:57:09.748 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-04-27T03:57:09.748 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000107979 s, 4.7 MB/s 2024-04-27T03:57:09.749 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-27T03:57:09.806 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-04-27T03:57:09.806 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/scratch_devs of=/dev/stdout 2024-04-27T03:57:09.824 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-27T03:57:09.825 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_1 2024-04-27T03:57:09.882 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-27T03:57:09.883 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T03:57:09.883 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-27T03:57:09.883 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T03:57:09.883 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T03:57:09.883 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-04-27 03:56:56.368967156 +0000 2024-04-27T03:57:09.883 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-04-27 03:50:30.021440725 +0000 2024-04-27T03:57:09.883 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-04-27 03:50:30.021440725 +0000 2024-04-27T03:57:09.883 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-04-27 03:50:30.021440725 +0000 2024-04-27T03:57:09.883 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-27T03:57:09.949 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-04-27T03:57:09.950 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-04-27T03:57:09.950 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000233851 s, 2.2 MB/s 2024-04-27T03:57:09.951 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-27T03:57:10.009 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_2 2024-04-27T03:57:10.067 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-27T03:57:10.067 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T03:57:10.067 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-27T03:57:10.067 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T03:57:10.067 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T03:57:10.067 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-04-27 03:56:56.369967133 +0000 2024-04-27T03:57:10.068 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-04-27 03:50:30.362432146 +0000 2024-04-27T03:57:10.068 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-04-27 03:50:30.362432146 +0000 2024-04-27T03:57:10.068 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-04-27 03:50:30.362432146 +0000 2024-04-27T03:57:10.068 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-27T03:57:10.133 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-04-27T03:57:10.133 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-04-27T03:57:10.133 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000221432 s, 2.3 MB/s 2024-04-27T03:57:10.135 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-27T03:57:10.193 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_3 2024-04-27T03:57:10.249 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-27T03:57:10.250 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T03:57:10.250 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-27T03:57:10.250 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T03:57:10.250 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T03:57:10.250 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-04-27 03:56:56.369967133 +0000 2024-04-27T03:57:10.250 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-04-27 03:50:30.704423541 +0000 2024-04-27T03:57:10.250 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-04-27 03:50:30.704423541 +0000 2024-04-27T03:57:10.250 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-04-27 03:50:30.704423541 +0000 2024-04-27T03:57:10.250 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-27T03:57:10.314 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-04-27T03:57:10.314 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-04-27T03:57:10.314 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000217043 s, 2.4 MB/s 2024-04-27T03:57:10.316 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-27T03:57:10.372 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_4 2024-04-27T03:57:10.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:10 smithi046 ceph-mon[26899]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:10.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:10 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:10.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:10 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:10.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:10 smithi046 ceph-mon[26899]: Reconfiguring mgr.smithi046.qrfdvr (unknown last config time)... 2024-04-27T03:57:10.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:10 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi046.qrfdvr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T03:57:10.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:10 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T03:57:10.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:10 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:10.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:10 smithi046 ceph-mon[26899]: Reconfiguring daemon mgr.smithi046.qrfdvr on smithi046 2024-04-27T03:57:10.428 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-27T03:57:10.429 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T03:57:10.429 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-27T03:57:10.429 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T03:57:10.429 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T03:57:10.429 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-04-27 03:56:56.370967110 +0000 2024-04-27T03:57:10.429 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-04-27 03:50:31.044414986 +0000 2024-04-27T03:57:10.429 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-04-27 03:50:31.044414986 +0000 2024-04-27T03:57:10.429 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-04-27 03:50:31.044414986 +0000 2024-04-27T03:57:10.429 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-27T03:57:10.492 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-04-27T03:57:10.492 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-04-27T03:57:10.492 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000239843 s, 2.1 MB/s 2024-04-27T03:57:10.494 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-27T03:57:10.550 INFO:tasks.cephadm:Deploying osd.0 on smithi046 with /dev/vg_nvme/lv_4... 2024-04-27T03:57:10.550 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-27T03:57:10.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:10 smithi073 ceph-mon[34160]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:10.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:10 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:10.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:10 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:10.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:10 smithi073 ceph-mon[34160]: Reconfiguring mgr.smithi046.qrfdvr (unknown last config time)... 2024-04-27T03:57:10.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:10 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi046.qrfdvr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T03:57:10.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:10 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T03:57:10.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:10 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:10.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:10 smithi073 ceph-mon[34160]: Reconfiguring daemon mgr.smithi046.qrfdvr on smithi046 2024-04-27T03:57:10.842 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T03:57:12.121 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:11 smithi046 ceph-mon[26899]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:12.121 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:11 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:12.121 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:11 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:12.121 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:11 smithi046 ceph-mon[26899]: Reconfiguring ceph-exporter.smithi046 (monmap changed)... 2024-04-27T03:57:12.121 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:11 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi046", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-27T03:57:12.121 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:11 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:12.121 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:11 smithi046 ceph-mon[26899]: Reconfiguring daemon ceph-exporter.smithi046 on smithi046 2024-04-27T03:57:12.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:11 smithi073 ceph-mon[34160]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:12.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:11 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:12.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:11 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:12.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:11 smithi073 ceph-mon[34160]: Reconfiguring ceph-exporter.smithi046 (monmap changed)... 2024-04-27T03:57:12.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:11 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi046", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-27T03:57:12.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:11 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:12.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:11 smithi073 ceph-mon[34160]: Reconfiguring daemon ceph-exporter.smithi046 on smithi046 2024-04-27T03:57:13.345 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-27T03:57:13.345 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-27T03:57:13.345 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-04-27T03:57:13.345 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-04-27T03:57:13.345 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00835178 s, 1.3 GB/s 2024-04-27T03:57:13.345 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-04-27T03:57:13.936 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi046:vg_nvme/lv_4 2024-04-27T03:57:14.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:14 smithi046 ceph-mon[26899]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:14.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:14.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:14.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:14 smithi046 ceph-mon[26899]: Reconfiguring crash.smithi046 (monmap changed)... 2024-04-27T03:57:14.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi046", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T03:57:14.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:14.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:14 smithi046 ceph-mon[26899]: Reconfiguring daemon crash.smithi046 on smithi046 2024-04-27T03:57:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:14 smithi073 ceph-mon[34160]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:14 smithi073 ceph-mon[34160]: Reconfiguring crash.smithi046 (monmap changed)... 2024-04-27T03:57:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi046", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T03:57:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:14 smithi073 ceph-mon[34160]: Reconfiguring daemon crash.smithi046 on smithi046 2024-04-27T03:57:15.112 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T03:57:16.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:16 smithi046 ceph-mon[26899]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:16.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:16 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:16.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:16 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:16.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:16 smithi046 ceph-mon[26899]: Reconfiguring alertmanager.smithi046 (dependencies changed)... 2024-04-27T03:57:16.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:16 smithi046 ceph-mon[26899]: Reconfiguring daemon alertmanager.smithi046 on smithi046 2024-04-27T03:57:16.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:16 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:57:16.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:16 smithi073 ceph-mon[34160]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:16.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:16 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:16.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:16 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:16.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:16 smithi073 ceph-mon[34160]: Reconfiguring alertmanager.smithi046 (dependencies changed)... 2024-04-27T03:57:16.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:16 smithi073 ceph-mon[34160]: Reconfiguring daemon alertmanager.smithi046 on smithi046 2024-04-27T03:57:16.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:16 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:57:18.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:18 smithi073 ceph-mon[34160]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:18.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:18 smithi046 ceph-mon[26899]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:19.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.279+0000 7f8f069fa700 1 -- 172.21.15.46:0/3237201460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f00101e40 msgr2=0x7f8f001022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:57:19.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.279+0000 7f8f069fa700 1 --2- 172.21.15.46:0/3237201460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f00101e40 0x7f8f001022a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4009910 tx=0x7f8ef4009c20 comp rx=0 tx=0).stop 2024-04-27T03:57:19.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.279+0000 7f8f069fa700 1 -- 172.21.15.46:0/3237201460 shutdown_connections 2024-04-27T03:57:19.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.279+0000 7f8f069fa700 1 --2- 172.21.15.46:0/3237201460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f00101e40 0x7f8f001022a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:19.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.279+0000 7f8f069fa700 1 --2- 172.21.15.46:0/3237201460 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f00100c40 0x7f8f00101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:19.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.279+0000 7f8f069fa700 1 -- 172.21.15.46:0/3237201460 >> 172.21.15.46:0/3237201460 conn(0x7f8f000fc200 msgr2=0x7f8f000fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:57:19.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.279+0000 7f8f069fa700 1 -- 172.21.15.46:0/3237201460 shutdown_connections 2024-04-27T03:57:19.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.279+0000 7f8f069fa700 1 -- 172.21.15.46:0/3237201460 wait complete. 2024-04-27T03:57:19.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.279+0000 7f8f069fa700 1 Processor -- start 2024-04-27T03:57:19.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.279+0000 7f8f069fa700 1 -- start start 2024-04-27T03:57:19.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.280+0000 7f8f069fa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f00100c40 0x7f8f00196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:57:19.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.280+0000 7f8f069fa700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f00101e40 0x7f8f00196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:57:19.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.280+0000 7f8f069fa700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f00197160 con 0x7f8f00100c40 2024-04-27T03:57:19.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.280+0000 7f8f069fa700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f001972a0 con 0x7f8f00101e40 2024-04-27T03:57:19.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.280+0000 7f8efffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f00100c40 0x7f8f00196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:57:19.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.280+0000 7f8efffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f00100c40 0x7f8f00196600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38778/0 (socket says 172.21.15.46:38778) 2024-04-27T03:57:19.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.280+0000 7f8efffff700 1 -- 172.21.15.46:0/223009342 learned_addr learned my addr 172.21.15.46:0/223009342 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:57:19.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.280+0000 7f8eff7fe700 1 --2- 172.21.15.46:0/223009342 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f00101e40 0x7f8f00196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:57:19.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.280+0000 7f8efffff700 1 -- 172.21.15.46:0/223009342 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f00101e40 msgr2=0x7f8f00196b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:57:19.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.280+0000 7f8efffff700 1 --2- 172.21.15.46:0/223009342 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f00101e40 0x7f8f00196b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:19.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.280+0000 7f8efffff700 1 -- 172.21.15.46:0/223009342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef40095f0 con 0x7f8f00100c40 2024-04-27T03:57:19.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.280+0000 7f8efffff700 1 --2- 172.21.15.46:0/223009342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f00100c40 0x7f8f00196600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000d900 tx=0x7f8ef000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:57:19.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.281+0000 7f8efd7fa700 1 -- 172.21.15.46:0/223009342 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ef00041f0 con 0x7f8f00100c40 2024-04-27T03:57:19.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.281+0000 7f8efd7fa700 1 -- 172.21.15.46:0/223009342 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ef0004350 con 0x7f8f00100c40 2024-04-27T03:57:19.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.281+0000 7f8f069fa700 1 -- 172.21.15.46:0/223009342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0019bd00 con 0x7f8f00100c40 2024-04-27T03:57:19.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.281+0000 7f8efd7fa700 1 -- 172.21.15.46:0/223009342 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ef0003cf0 con 0x7f8f00100c40 2024-04-27T03:57:19.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.281+0000 7f8f069fa700 1 -- 172.21.15.46:0/223009342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0019c270 con 0x7f8f00100c40 2024-04-27T03:57:19.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.282+0000 7f8f069fa700 1 -- 172.21.15.46:0/223009342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f001907c0 con 0x7f8f00100c40 2024-04-27T03:57:19.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.282+0000 7f8efd7fa700 1 -- 172.21.15.46:0/223009342 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7f8ef0009730 con 0x7f8f00100c40 2024-04-27T03:57:19.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.283+0000 7f8efd7fa700 1 --2- 172.21.15.46:0/223009342 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8ee806cc60 0x7f8ee806f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:57:19.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.283+0000 7f8efd7fa700 1 -- 172.21.15.46:0/223009342 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f8ef008afd0 con 0x7f8f00100c40 2024-04-27T03:57:19.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.283+0000 7f8eff7fe700 1 --2- 172.21.15.46:0/223009342 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8ee806cc60 0x7f8ee806f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:57:19.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.283+0000 7f8eff7fe700 1 --2- 172.21.15.46:0/223009342 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8ee806cc60 0x7f8ee806f120 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4005090 tx=0x7f8ef4005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:57:19.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.288+0000 7f8efd7fa700 1 -- 172.21.15.46:0/223009342 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f8ef0059f10 con 0x7f8f00100c40 2024-04-27T03:57:19.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:19.512+0000 7f8f069fa700 1 -- 172.21.15.46:0/223009342 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f0002cf00 con 0x7f8ee806cc60 2024-04-27T03:57:20.334 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:20 smithi046 ceph-mon[26899]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:20.334 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:20 smithi046 ceph-mon[26899]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:57:20.334 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:20 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T03:57:20.334 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:20 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T03:57:20.335 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:20 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:20.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:20 smithi073 ceph-mon[34160]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:20.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:20 smithi073 ceph-mon[34160]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:57:20.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:20 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T03:57:20.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:20 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T03:57:20.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:20 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:22.406 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:22 smithi046 ceph-mon[26899]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:22.406 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:22 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:22.406 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:22 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:22.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:22 smithi073 ceph-mon[34160]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:22.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:22 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:22.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:22 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:23.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:23 smithi046 ceph-mon[26899]: Reconfiguring grafana.smithi046 (dependencies changed)... 2024-04-27T03:57:23.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:23 smithi046 ceph-mon[26899]: Reconfiguring daemon grafana.smithi046 on smithi046 2024-04-27T03:57:23.478 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:23 smithi073 ceph-mon[34160]: Reconfiguring grafana.smithi046 (dependencies changed)... 2024-04-27T03:57:23.478 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:23 smithi073 ceph-mon[34160]: Reconfiguring daemon grafana.smithi046 on smithi046 2024-04-27T03:57:24.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:24 smithi046 ceph-mon[26899]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:24.186 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:24 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/1685200440' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0310053-0895-4608-9021-6c04075a1f9c"}]: dispatch 2024-04-27T03:57:24.186 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:24 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/1685200440' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0310053-0895-4608-9021-6c04075a1f9c"}]': finished 2024-04-27T03:57:24.186 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:24 smithi046 ceph-mon[26899]: osdmap e6: 1 total, 0 up, 1 in 2024-04-27T03:57:24.186 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:24 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T03:57:24.186 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:24 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/3161904275' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T03:57:24.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:24 smithi073 ceph-mon[34160]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:24.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:24 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/1685200440' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0310053-0895-4608-9021-6c04075a1f9c"}]: dispatch 2024-04-27T03:57:24.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:24 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/1685200440' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0310053-0895-4608-9021-6c04075a1f9c"}]': finished 2024-04-27T03:57:24.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:24 smithi073 ceph-mon[34160]: osdmap e6: 1 total, 0 up, 1 in 2024-04-27T03:57:24.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:24 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T03:57:24.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:24 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/3161904275' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T03:57:26.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:26 smithi046 ceph-mon[26899]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:26.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:26 smithi073 ceph-mon[34160]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:28.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:28 smithi046 ceph-mon[26899]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:28.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:28 smithi073 ceph-mon[34160]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:30.398 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:30 smithi046 ceph-mon[26899]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:30.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:30 smithi073 ceph-mon[34160]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:31.386 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:57:31.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:31 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:57:32.360 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:32 smithi046 ceph-mon[26899]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:32.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:32 smithi073 ceph-mon[34160]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:33.332 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:33 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:33.333 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:33 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:33.333 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:33 smithi046 ceph-mon[26899]: Reconfiguring prometheus.smithi046 (dependencies changed)... 2024-04-27T03:57:33.333 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:33 smithi046 ceph-mon[26899]: Reconfiguring daemon prometheus.smithi046 on smithi046 2024-04-27T03:57:33.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:33 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:33.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:33 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:33.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:33 smithi073 ceph-mon[34160]: Reconfiguring prometheus.smithi046 (dependencies changed)... 2024-04-27T03:57:33.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:33 smithi073 ceph-mon[34160]: Reconfiguring daemon prometheus.smithi046 on smithi046 2024-04-27T03:57:34.579 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:34 smithi073 ceph-mon[34160]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:34.579 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:34 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T03:57:34.579 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:34 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:34.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:34 smithi046 ceph-mon[26899]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:34.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:34 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T03:57:34.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:34 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:35.463 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:35 smithi046 ceph-mon[26899]: Deploying daemon osd.0 on smithi046 2024-04-27T03:57:35.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:35 smithi073 ceph-mon[34160]: Deploying daemon osd.0 on smithi046 2024-04-27T03:57:36.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:36 smithi046 ceph-mon[26899]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:36.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:36 smithi073 ceph-mon[34160]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:38.563 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:38 smithi073 ceph-mon[34160]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:38.563 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:38 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:38.563 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:38 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:38.563 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:38 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi073", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-27T03:57:38.563 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:38 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:38.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:38 smithi046 ceph-mon[26899]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:38.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:38 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:38.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:38 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:38.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:38 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi073", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-27T03:57:38.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:38 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:39.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:39 smithi073 ceph-mon[34160]: Reconfiguring ceph-exporter.smithi073 (monmap changed)... 2024-04-27T03:57:39.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:39 smithi073 ceph-mon[34160]: Reconfiguring daemon ceph-exporter.smithi073 on smithi073 2024-04-27T03:57:39.669 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:39 smithi046 ceph-mon[26899]: Reconfiguring ceph-exporter.smithi073 (monmap changed)... 2024-04-27T03:57:39.669 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:39 smithi046 ceph-mon[26899]: Reconfiguring daemon ceph-exporter.smithi073 on smithi073 2024-04-27T03:57:40.438 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:40 smithi073 ceph-mon[34160]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:40.438 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:40 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:40.438 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:40 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:40.438 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:40 smithi073 ceph-mon[34160]: Reconfiguring crash.smithi073 (monmap changed)... 2024-04-27T03:57:40.438 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:40 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T03:57:40.438 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:40 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:40.438 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:40 smithi073 ceph-mon[34160]: Reconfiguring daemon crash.smithi073 on smithi073 2024-04-27T03:57:40.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:40 smithi046 ceph-mon[26899]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:40.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:40 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:40.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:40 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:40.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:40 smithi046 ceph-mon[26899]: Reconfiguring crash.smithi073 (monmap changed)... 2024-04-27T03:57:40.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:40 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T03:57:40.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:40 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:40.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:40 smithi046 ceph-mon[26899]: Reconfiguring daemon crash.smithi073 on smithi073 2024-04-27T03:57:41.918 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:41 smithi046 ceph-mon[26899]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:41.919 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:41 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:41.919 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:41 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:41.919 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:41 smithi046 ceph-mon[26899]: Reconfiguring mgr.smithi073.awyllt (monmap changed)... 2024-04-27T03:57:41.919 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:41 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi073.awyllt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T03:57:41.919 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:41 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T03:57:41.919 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:41 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:41.919 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:41 smithi046 ceph-mon[26899]: Reconfiguring daemon mgr.smithi073.awyllt on smithi073 2024-04-27T03:57:42.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:41 smithi073 ceph-mon[34160]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:42.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:41 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:42.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:41 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:42.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:41 smithi073 ceph-mon[34160]: Reconfiguring mgr.smithi073.awyllt (monmap changed)... 2024-04-27T03:57:42.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:41 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi073.awyllt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T03:57:42.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:41 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T03:57:42.063 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:41 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:42.063 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:41 smithi073 ceph-mon[34160]: Reconfiguring daemon mgr.smithi073.awyllt on smithi073 2024-04-27T03:57:43.381 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:43 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:43.381 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:43 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:43.381 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:43 smithi046 ceph-mon[26899]: Reconfiguring mon.smithi073 (monmap changed)... 2024-04-27T03:57:43.381 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:43 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T03:57:43.382 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:43 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T03:57:43.382 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:43 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:43.382 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:43 smithi046 ceph-mon[26899]: Reconfiguring daemon mon.smithi073 on smithi073 2024-04-27T03:57:43.382 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:43 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:43.382 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:43 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:43.443 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:43 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:43.443 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:43 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:43.443 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:43 smithi073 ceph-mon[34160]: Reconfiguring mon.smithi073 (monmap changed)... 2024-04-27T03:57:43.444 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:43 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T03:57:43.444 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:43 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T03:57:43.444 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:43 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:43.444 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:43 smithi073 ceph-mon[34160]: Reconfiguring daemon mon.smithi073 on smithi073 2024-04-27T03:57:43.444 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:43 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:43.444 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:43 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:44.544 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:44.544 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:44.544 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:44.545 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T03:57:44.545 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T03:57:44.545 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi046:9093"}]: dispatch 2024-04-27T03:57:44.545 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi046:9093"}]: dispatch 2024-04-27T03:57:44.545 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:44.545 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T03:57:44.545 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T03:57:44.545 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi046:3000"}]: dispatch 2024-04-27T03:57:44.545 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi046:3000"}]: dispatch 2024-04-27T03:57:44.545 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:44.545 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T03:57:44.545 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T03:57:44.545 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi046:9095"}]: dispatch 2024-04-27T03:57:44.546 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi046:9095"}]: dispatch 2024-04-27T03:57:44.546 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:44.546 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:57:44.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T03:57:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T03:57:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi046:9093"}]: dispatch 2024-04-27T03:57:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi046:9093"}]: dispatch 2024-04-27T03:57:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T03:57:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T03:57:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi046:3000"}]: dispatch 2024-04-27T03:57:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi046:3000"}]: dispatch 2024-04-27T03:57:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T03:57:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T03:57:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi046:9095"}]: dispatch 2024-04-27T03:57:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi046:9095"}]: dispatch 2024-04-27T03:57:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:57:46.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:46.105+0000 7f8efd7fa700 1 -- 172.21.15.46:0/223009342 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8f0002cf00 con 0x7f8ee806cc60 2024-04-27T03:57:46.107 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 0 on host 'smithi046' 2024-04-27T03:57:46.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:46.107+0000 7f8f069fa700 1 -- 172.21.15.46:0/223009342 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8ee806cc60 msgr2=0x7f8ee806f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:57:46.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:46.107+0000 7f8f069fa700 1 --2- 172.21.15.46:0/223009342 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8ee806cc60 0x7f8ee806f120 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4005090 tx=0x7f8ef4005960 comp rx=0 tx=0).stop 2024-04-27T03:57:46.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:46.107+0000 7f8f069fa700 1 -- 172.21.15.46:0/223009342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f00100c40 msgr2=0x7f8f00196600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:57:46.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:46.107+0000 7f8f069fa700 1 --2- 172.21.15.46:0/223009342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f00100c40 0x7f8f00196600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000d900 tx=0x7f8ef000dc10 comp rx=0 tx=0).stop 2024-04-27T03:57:46.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:46.108+0000 7f8f069fa700 1 -- 172.21.15.46:0/223009342 shutdown_connections 2024-04-27T03:57:46.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:46.108+0000 7f8f069fa700 1 --2- 172.21.15.46:0/223009342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f00100c40 0x7f8f00196600 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:46.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:46.108+0000 7f8f069fa700 1 --2- 172.21.15.46:0/223009342 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8ee806cc60 0x7f8ee806f120 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:46.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:46.108+0000 7f8f069fa700 1 --2- 172.21.15.46:0/223009342 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f00101e40 0x7f8f00196b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:46.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:46.108+0000 7f8f069fa700 1 -- 172.21.15.46:0/223009342 >> 172.21.15.46:0/223009342 conn(0x7f8f000fc200 msgr2=0x7f8f001053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:57:46.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:46.108+0000 7f8f069fa700 1 -- 172.21.15.46:0/223009342 shutdown_connections 2024-04-27T03:57:46.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:46.108+0000 7f8f069fa700 1 -- 172.21.15.46:0/223009342 wait complete. 2024-04-27T03:57:46.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:46 smithi073 ceph-mon[34160]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:46.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:46 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:46.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:46 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:46.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:46 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:57:46.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:46 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:46.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:46 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:46.654 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:46 smithi046 ceph-mon[26899]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:46.655 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:46 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:46.655 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:46 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:46.655 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:46 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:57:46.655 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:46 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:46.655 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:46 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:46.759 DEBUG:teuthology.orchestra.run.smithi046:osd.0> sudo journalctl -f -n 0 -u ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@osd.0.service 2024-04-27T03:57:46.761 INFO:tasks.cephadm:Deploying osd.1 on smithi046 with /dev/vg_nvme/lv_3... 2024-04-27T03:57:46.762 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-27T03:57:47.461 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T03:57:47.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:47 smithi046 ceph-mon[26899]: from='osd.0 [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-27T03:57:47.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:47 smithi073 ceph-mon[34160]: from='osd.0 [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-27T03:57:48.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:48 smithi046 ceph-mon[26899]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:48.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:48 smithi046 ceph-mon[26899]: from='osd.0 [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-27T03:57:48.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:48 smithi046 ceph-mon[26899]: osdmap e7: 1 total, 0 up, 1 in 2024-04-27T03:57:48.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:48 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T03:57:48.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:48 smithi046 ceph-mon[26899]: from='osd.0 [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-27T03:57:48.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:48 smithi073 ceph-mon[34160]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:48.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:48 smithi073 ceph-mon[34160]: from='osd.0 [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-27T03:57:48.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:48 smithi073 ceph-mon[34160]: osdmap e7: 1 total, 0 up, 1 in 2024-04-27T03:57:48.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:48 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T03:57:48.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:48 smithi073 ceph-mon[34160]: from='osd.0 [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-27T03:57:49.411 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 03:57:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T03:57:49.163+0000 7f6d5fce9700 -1 osd.0 0 waiting for initial osdmap 2024-04-27T03:57:49.411 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 03:57:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T03:57:49.166+0000 7f6d582d7700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T03:57:49.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:49 smithi046 ceph-mon[26899]: from='osd.0 [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-27T03:57:49.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:49 smithi046 ceph-mon[26899]: osdmap e8: 1 total, 0 up, 1 in 2024-04-27T03:57:49.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:49 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T03:57:49.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:49 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T03:57:49.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:49 smithi046 ceph-mon[26899]: from='osd.0 [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853]' entity='osd.0' 2024-04-27T03:57:49.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:49 smithi073 ceph-mon[34160]: from='osd.0 [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-27T03:57:49.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:49 smithi073 ceph-mon[34160]: osdmap e8: 1 total, 0 up, 1 in 2024-04-27T03:57:49.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:49 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T03:57:49.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:49 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T03:57:49.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:49 smithi073 ceph-mon[34160]: from='osd.0 [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853]' entity='osd.0' 2024-04-27T03:57:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-27T03:57:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-27T03:57:50.018 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-04-27T03:57:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-04-27T03:57:50.018 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109613 s, 957 MB/s 2024-04-27T03:57:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-04-27T03:57:50.635 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:50 smithi046 ceph-mon[26899]: purged_snaps scrub starts 2024-04-27T03:57:50.635 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:50 smithi046 ceph-mon[26899]: purged_snaps scrub ok 2024-04-27T03:57:50.635 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:50 smithi046 ceph-mon[26899]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:50.635 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T03:57:50.635 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:50 smithi046 ceph-mon[26899]: osd.0 [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853] boot 2024-04-27T03:57:50.635 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:50 smithi046 ceph-mon[26899]: osdmap e9: 1 total, 1 up, 1 in 2024-04-27T03:57:50.635 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T03:57:50.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:50 smithi073 ceph-mon[34160]: purged_snaps scrub starts 2024-04-27T03:57:50.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:50 smithi073 ceph-mon[34160]: purged_snaps scrub ok 2024-04-27T03:57:50.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:50 smithi073 ceph-mon[34160]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T03:57:50.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:50 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T03:57:50.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:50 smithi073 ceph-mon[34160]: osd.0 [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853] boot 2024-04-27T03:57:50.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:50 smithi073 ceph-mon[34160]: osdmap e9: 1 total, 1 up, 1 in 2024-04-27T03:57:50.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:50 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T03:57:50.943 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi046:vg_nvme/lv_3 2024-04-27T03:57:51.634 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T03:57:52.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:52 smithi046 ceph-mon[26899]: pgmap v37: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:57:52.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:52 smithi046 ceph-mon[26899]: osdmap e10: 1 total, 1 up, 1 in 2024-04-27T03:57:52.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:52 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:52.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:52 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:52.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:52 smithi073 ceph-mon[34160]: pgmap v37: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:57:52.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:52 smithi073 ceph-mon[34160]: osdmap e10: 1 total, 1 up, 1 in 2024-04-27T03:57:52.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:52 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:52.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:52 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:53.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.222+0000 7f7b1c477700 1 -- 172.21.15.46:0/1334402741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b14101e40 msgr2=0x7f7b141022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:57:53.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.222+0000 7f7b1c477700 1 --2- 172.21.15.46:0/1334402741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b14101e40 0x7f7b141022a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7b04009910 tx=0x7f7b04009c20 comp rx=0 tx=0).stop 2024-04-27T03:57:53.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.222+0000 7f7b1c477700 1 -- 172.21.15.46:0/1334402741 shutdown_connections 2024-04-27T03:57:53.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.222+0000 7f7b1c477700 1 --2- 172.21.15.46:0/1334402741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b14101e40 0x7f7b141022a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:53.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.222+0000 7f7b1c477700 1 --2- 172.21.15.46:0/1334402741 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b14100c40 0x7f7b14101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:53.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.222+0000 7f7b1c477700 1 -- 172.21.15.46:0/1334402741 >> 172.21.15.46:0/1334402741 conn(0x7f7b140fc1e0 msgr2=0x7f7b140fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:57:53.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.222+0000 7f7b1c477700 1 -- 172.21.15.46:0/1334402741 shutdown_connections 2024-04-27T03:57:53.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.222+0000 7f7b1c477700 1 -- 172.21.15.46:0/1334402741 wait complete. 2024-04-27T03:57:53.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.222+0000 7f7b1c477700 1 Processor -- start 2024-04-27T03:57:53.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.223+0000 7f7b1c477700 1 -- start start 2024-04-27T03:57:53.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.223+0000 7f7b1c477700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b14100c40 0x7f7b14196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:57:53.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.223+0000 7f7b1c477700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b14101e40 0x7f7b14196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:57:53.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.223+0000 7f7b1c477700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b14197180 con 0x7f7b14100c40 2024-04-27T03:57:53.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.223+0000 7f7b1c477700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b141972c0 con 0x7f7b14101e40 2024-04-27T03:57:53.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.223+0000 7f7b1a213700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b14100c40 0x7f7b14196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:57:53.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.223+0000 7f7b1a213700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b14100c40 0x7f7b14196620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57792/0 (socket says 172.21.15.46:57792) 2024-04-27T03:57:53.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.223+0000 7f7b1a213700 1 -- 172.21.15.46:0/3307188892 learned_addr learned my addr 172.21.15.46:0/3307188892 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:57:53.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.223+0000 7f7b19a12700 1 --2- 172.21.15.46:0/3307188892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b14101e40 0x7f7b14196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:57:53.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.223+0000 7f7b1a213700 1 -- 172.21.15.46:0/3307188892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b14101e40 msgr2=0x7f7b14196b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:57:53.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.223+0000 7f7b1a213700 1 --2- 172.21.15.46:0/3307188892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b14101e40 0x7f7b14196b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:57:53.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.223+0000 7f7b1a213700 1 -- 172.21.15.46:0/3307188892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b040095f0 con 0x7f7b14100c40 2024-04-27T03:57:53.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.223+0000 7f7b19a12700 1 --2- 172.21.15.46:0/3307188892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b14101e40 0x7f7b14196b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T03:57:53.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.223+0000 7f7b1a213700 1 --2- 172.21.15.46:0/3307188892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b14100c40 0x7f7b14196620 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7b1000d900 tx=0x7f7b1000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:57:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.224+0000 7f7b0b7fe700 1 -- 172.21.15.46:0/3307188892 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b100041f0 con 0x7f7b14100c40 2024-04-27T03:57:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.224+0000 7f7b1c477700 1 -- 172.21.15.46:0/3307188892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b1419bd20 con 0x7f7b14100c40 2024-04-27T03:57:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.224+0000 7f7b0b7fe700 1 -- 172.21.15.46:0/3307188892 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b10004350 con 0x7f7b14100c40 2024-04-27T03:57:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.224+0000 7f7b0b7fe700 1 -- 172.21.15.46:0/3307188892 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b10003dc0 con 0x7f7b14100c40 2024-04-27T03:57:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.224+0000 7f7b1c477700 1 -- 172.21.15.46:0/3307188892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b1419c290 con 0x7f7b14100c40 2024-04-27T03:57:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.225+0000 7f7b1c477700 1 -- 172.21.15.46:0/3307188892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b141907c0 con 0x7f7b14100c40 2024-04-27T03:57:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.226+0000 7f7b0b7fe700 1 -- 172.21.15.46:0/3307188892 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7f7b10009730 con 0x7f7b14100c40 2024-04-27T03:57:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.226+0000 7f7b0b7fe700 1 --2- 172.21.15.46:0/3307188892 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7b0006cc60 0x7f7b0006f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:57:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.226+0000 7f7b0b7fe700 1 -- 172.21.15.46:0/3307188892 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7f7b1008b3b0 con 0x7f7b14100c40 2024-04-27T03:57:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.226+0000 7f7b19a12700 1 --2- 172.21.15.46:0/3307188892 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7b0006cc60 0x7f7b0006f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:57:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.227+0000 7f7b19a12700 1 --2- 172.21.15.46:0/3307188892 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7b0006cc60 0x7f7b0006f120 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7b040051c0 tx=0x7f7b04005a90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:57:53.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.231+0000 7f7b0b7fe700 1 -- 172.21.15.46:0/3307188892 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f7b1005a090 con 0x7f7b14100c40 2024-04-27T03:57:53.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:57:53.457+0000 7f7b1c477700 1 -- 172.21.15.46:0/3307188892 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b1402ced0 con 0x7f7b0006cc60 2024-04-27T03:57:54.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:54 smithi046 ceph-mon[26899]: pgmap v39: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:57:54.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:54 smithi046 ceph-mon[26899]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:57:54.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:54 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T03:57:54.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:54 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T03:57:54.533 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:54 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:54.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:54 smithi073 ceph-mon[34160]: pgmap v39: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:57:54.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:54 smithi073 ceph-mon[34160]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:57:54.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:54 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T03:57:54.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:54 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T03:57:54.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:54 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:56.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:56 smithi073 ceph-mon[34160]: pgmap v40: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:57:56.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:56 smithi046 ceph-mon[26899]: pgmap v40: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:57:58.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:58 smithi073 ceph-mon[34160]: pgmap v41: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:57:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:58 smithi073 ceph-mon[34160]: Detected new or changed devices on smithi046 2024-04-27T03:57:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:58 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:58 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:58 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:57:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:58 smithi073 ceph-mon[34160]: Adjusting osd_memory_target on smithi046 to 11845M 2024-04-27T03:57:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:58 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:58 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:58 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:57:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:58 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:58 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:57:58.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:58 smithi046 ceph-mon[26899]: pgmap v41: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:57:58.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:58 smithi046 ceph-mon[26899]: Detected new or changed devices on smithi046 2024-04-27T03:57:58.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:58 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:58 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:58 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:57:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:58 smithi046 ceph-mon[26899]: Adjusting osd_memory_target on smithi046 to 11845M 2024-04-27T03:57:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:58 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:58 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:57:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:58 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:57:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:58 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:58 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:57:59.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:59 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/3928513976' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57d1cc25-d5db-41fd-aa72-e380cedf2225"}]: dispatch 2024-04-27T03:57:59.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:59 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/3928513976' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57d1cc25-d5db-41fd-aa72-e380cedf2225"}]': finished 2024-04-27T03:57:59.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:59 smithi046 ceph-mon[26899]: osdmap e11: 2 total, 1 up, 2 in 2024-04-27T03:57:59.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:59 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T03:57:59.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:59 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/3737460970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T03:57:59.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:59 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:59.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:57:59 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:59.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:59 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/3928513976' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57d1cc25-d5db-41fd-aa72-e380cedf2225"}]: dispatch 2024-04-27T03:57:59.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:59 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/3928513976' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57d1cc25-d5db-41fd-aa72-e380cedf2225"}]': finished 2024-04-27T03:57:59.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:59 smithi073 ceph-mon[34160]: osdmap e11: 2 total, 1 up, 2 in 2024-04-27T03:57:59.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:59 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T03:57:59.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:59 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/3737460970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T03:57:59.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:59 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:57:59.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:57:59 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:00.562 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:00 smithi046 ceph-mon[26899]: pgmap v43: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:00.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:00 smithi073 ceph-mon[34160]: pgmap v43: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:01.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:01 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:58:01.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:01 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:58:02.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:02 smithi046 ceph-mon[26899]: pgmap v44: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:02.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:02 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:02.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:02 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:02.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:02 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:02.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:02 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:58:02.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:02 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:02.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:02 smithi073 ceph-mon[34160]: pgmap v44: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:02.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:02 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:02.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:02 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:02.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:02 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:02.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:02 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:58:02.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:02 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:04.634 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:04 smithi046 ceph-mon[26899]: pgmap v45: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:04.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:04 smithi073 ceph-mon[34160]: pgmap v45: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:06.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:06 smithi073 ceph-mon[34160]: pgmap v46: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:06.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:06 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T03:58:06.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:06 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:06.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:06 smithi073 ceph-mon[34160]: Deploying daemon osd.1 on smithi046 2024-04-27T03:58:06.875 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:06 smithi046 ceph-mon[26899]: pgmap v46: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:06.876 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:06 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T03:58:06.876 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:06 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:06.876 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:06 smithi046 ceph-mon[26899]: Deploying daemon osd.1 on smithi046 2024-04-27T03:58:08.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:08 smithi073 ceph-mon[34160]: pgmap v47: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:08.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:08 smithi046 ceph-mon[26899]: pgmap v47: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:10.783 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:10 smithi046 ceph-mon[26899]: pgmap v48: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:10.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:10 smithi073 ceph-mon[34160]: pgmap v48: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:11.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:11 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:58:11.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:11 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:11.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:11 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:11.893 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:11 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:58:11.893 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:11 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:11.893 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:11 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:12.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:12 smithi073 ceph-mon[34160]: pgmap v49: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:12.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:12 smithi046 ceph-mon[26899]: pgmap v49: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:13.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:13 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:13.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:13 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:13.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:13 smithi046 ceph-mon[26899]: pgmap v50: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:14.067 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 1 on host 'smithi046' 2024-04-27T03:58:14.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:14.065+0000 7f7b0b7fe700 1 -- 172.21.15.46:0/3307188892 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7b1402ced0 con 0x7f7b0006cc60 2024-04-27T03:58:14.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:14.068+0000 7f7b1c477700 1 -- 172.21.15.46:0/3307188892 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7b0006cc60 msgr2=0x7f7b0006f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:58:14.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:14.068+0000 7f7b1c477700 1 --2- 172.21.15.46:0/3307188892 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7b0006cc60 0x7f7b0006f120 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7b040051c0 tx=0x7f7b04005a90 comp rx=0 tx=0).stop 2024-04-27T03:58:14.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:14.068+0000 7f7b1c477700 1 -- 172.21.15.46:0/3307188892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b14100c40 msgr2=0x7f7b14196620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:58:14.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:14.068+0000 7f7b1c477700 1 --2- 172.21.15.46:0/3307188892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b14100c40 0x7f7b14196620 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7b1000d900 tx=0x7f7b1000dcc0 comp rx=0 tx=0).stop 2024-04-27T03:58:14.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:14.068+0000 7f7b1c477700 1 -- 172.21.15.46:0/3307188892 shutdown_connections 2024-04-27T03:58:14.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:14.068+0000 7f7b1c477700 1 --2- 172.21.15.46:0/3307188892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b14100c40 0x7f7b14196620 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:58:14.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:14.068+0000 7f7b1c477700 1 --2- 172.21.15.46:0/3307188892 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7b0006cc60 0x7f7b0006f120 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:58:14.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:14.068+0000 7f7b1c477700 1 --2- 172.21.15.46:0/3307188892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b14101e40 0x7f7b14196b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:58:14.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:14.068+0000 7f7b1c477700 1 -- 172.21.15.46:0/3307188892 >> 172.21.15.46:0/3307188892 conn(0x7f7b140fc1e0 msgr2=0x7f7b14105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:58:14.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:14.068+0000 7f7b1c477700 1 -- 172.21.15.46:0/3307188892 shutdown_connections 2024-04-27T03:58:14.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:14.068+0000 7f7b1c477700 1 -- 172.21.15.46:0/3307188892 wait complete. 2024-04-27T03:58:14.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:13 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:14.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:13 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:14.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:13 smithi073 ceph-mon[34160]: pgmap v50: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:14.695 DEBUG:teuthology.orchestra.run.smithi046:osd.1> sudo journalctl -f -n 0 -u ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@osd.1.service 2024-04-27T03:58:14.697 INFO:tasks.cephadm:Deploying osd.2 on smithi046 with /dev/vg_nvme/lv_2... 2024-04-27T03:58:14.697 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-27T03:58:14.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 03:58:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T03:58:14.747+0000 7f71f43947c0 -1 osd.1 0 log_to_monitors true 2024-04-27T03:58:15.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:15 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:15.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:15 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:15.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:15 smithi073 ceph-mon[34160]: from='osd.1 [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-27T03:58:15.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:15 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:15.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:15 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:15.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:15 smithi046 ceph-mon[26899]: from='osd.1 [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-27T03:58:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T03:58:16.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:16 smithi073 ceph-mon[34160]: pgmap v51: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:16.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:16 smithi073 ceph-mon[34160]: from='osd.1 [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-27T03:58:16.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:16 smithi073 ceph-mon[34160]: osdmap e12: 2 total, 1 up, 2 in 2024-04-27T03:58:16.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:16 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T03:58:16.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:16 smithi073 ceph-mon[34160]: from='osd.1 [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-27T03:58:16.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:16 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:58:16.377 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 03:58:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T03:58:16.055+0000 7f71e91f8700 -1 osd.1 0 waiting for initial osdmap 2024-04-27T03:58:16.377 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 03:58:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T03:58:16.059+0000 7f71e57ee700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T03:58:16.377 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:16 smithi046 ceph-mon[26899]: pgmap v51: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:16.377 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:16 smithi046 ceph-mon[26899]: from='osd.1 [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-27T03:58:16.377 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:16 smithi046 ceph-mon[26899]: osdmap e12: 2 total, 1 up, 2 in 2024-04-27T03:58:16.377 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:16 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T03:58:16.377 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:16 smithi046 ceph-mon[26899]: from='osd.1 [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-27T03:58:16.377 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:16 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:58:17.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:17 smithi046 ceph-mon[26899]: from='osd.1 [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-27T03:58:17.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:17 smithi046 ceph-mon[26899]: osdmap e13: 2 total, 1 up, 2 in 2024-04-27T03:58:17.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:17 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T03:58:17.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:17 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T03:58:17.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:17 smithi073 ceph-mon[34160]: from='osd.1 [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-27T03:58:17.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:17 smithi073 ceph-mon[34160]: osdmap e13: 2 total, 1 up, 2 in 2024-04-27T03:58:17.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:17 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T03:58:17.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:17 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T03:58:18.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:18 smithi073 ceph-mon[34160]: purged_snaps scrub starts 2024-04-27T03:58:18.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:18 smithi073 ceph-mon[34160]: purged_snaps scrub ok 2024-04-27T03:58:18.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:18 smithi073 ceph-mon[34160]: pgmap v54: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:18.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:18 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T03:58:18.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:18 smithi073 ceph-mon[34160]: osd.1 [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418] boot 2024-04-27T03:58:18.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:18 smithi073 ceph-mon[34160]: osdmap e14: 2 total, 2 up, 2 in 2024-04-27T03:58:18.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:18 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T03:58:18.338 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:18 smithi046 ceph-mon[26899]: purged_snaps scrub starts 2024-04-27T03:58:18.338 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:18 smithi046 ceph-mon[26899]: purged_snaps scrub ok 2024-04-27T03:58:18.338 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:18 smithi046 ceph-mon[26899]: pgmap v54: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-27T03:58:18.338 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:18 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T03:58:18.338 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:18 smithi046 ceph-mon[26899]: osd.1 [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418] boot 2024-04-27T03:58:18.338 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:18 smithi046 ceph-mon[26899]: osdmap e14: 2 total, 2 up, 2 in 2024-04-27T03:58:18.338 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:18 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T03:58:18.977 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-27T03:58:18.977 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-27T03:58:18.977 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-04-27T03:58:18.977 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-04-27T03:58:18.978 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00890133 s, 1.2 GB/s 2024-04-27T03:58:18.978 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-04-27T03:58:19.306 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:19 smithi046 ceph-mon[26899]: osdmap e15: 2 total, 2 up, 2 in 2024-04-27T03:58:19.306 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:19 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:19.306 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:19 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:19.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:19 smithi073 ceph-mon[34160]: osdmap e15: 2 total, 2 up, 2 in 2024-04-27T03:58:19.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:19 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:19.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:19 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:19.578 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi046:vg_nvme/lv_2 2024-04-27T03:58:20.126 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:20 smithi046 ceph-mon[26899]: pgmap v57: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:20.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:20 smithi073 ceph-mon[34160]: pgmap v57: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T03:58:22.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:22 smithi046 ceph-mon[26899]: pgmap v58: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:22.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:22 smithi073 ceph-mon[34160]: pgmap v58: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:23.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.208+0000 7f6798cdc700 1 -- 172.21.15.46:0/747533233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6794100d90 msgr2=0x7f67941011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:58:23.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.208+0000 7f6798cdc700 1 --2- 172.21.15.46:0/747533233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6794100d90 0x7f67941011f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6788009910 tx=0x7f6788009c20 comp rx=0 tx=0).stop 2024-04-27T03:58:23.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.208+0000 7f6798cdc700 1 -- 172.21.15.46:0/747533233 shutdown_connections 2024-04-27T03:58:23.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.208+0000 7f6798cdc700 1 --2- 172.21.15.46:0/747533233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6794100d90 0x7f67941011f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:58:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.208+0000 7f6798cdc700 1 --2- 172.21.15.46:0/747533233 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67940ffb90 0x7f67940fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:58:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.208+0000 7f6798cdc700 1 -- 172.21.15.46:0/747533233 >> 172.21.15.46:0/747533233 conn(0x7f67940fb110 msgr2=0x7f67940fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:58:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.208+0000 7f6798cdc700 1 -- 172.21.15.46:0/747533233 shutdown_connections 2024-04-27T03:58:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.208+0000 7f6798cdc700 1 -- 172.21.15.46:0/747533233 wait complete. 2024-04-27T03:58:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.209+0000 7f6798cdc700 1 Processor -- start 2024-04-27T03:58:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.209+0000 7f6798cdc700 1 -- start start 2024-04-27T03:58:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.209+0000 7f6798cdc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67940ffb90 0x7f6794196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:58:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.209+0000 7f6798cdc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6794100d90 0x7f6794196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:58:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.209+0000 7f6798cdc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6794197270 con 0x7f67940ffb90 2024-04-27T03:58:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.209+0000 7f6798cdc700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67941973b0 con 0x7f6794100d90 2024-04-27T03:58:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.209+0000 7f679259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67940ffb90 0x7f6794196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:58:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.209+0000 7f679259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67940ffb90 0x7f6794196710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40356/0 (socket says 172.21.15.46:40356) 2024-04-27T03:58:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.209+0000 7f679259c700 1 -- 172.21.15.46:0/3997002224 learned_addr learned my addr 172.21.15.46:0/3997002224 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T03:58:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.210+0000 7f6791d9b700 1 --2- 172.21.15.46:0/3997002224 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6794100d90 0x7f6794196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:58:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.210+0000 7f679259c700 1 -- 172.21.15.46:0/3997002224 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6794100d90 msgr2=0x7f6794196c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:58:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.210+0000 7f679259c700 1 --2- 172.21.15.46:0/3997002224 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6794100d90 0x7f6794196c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:58:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.210+0000 7f679259c700 1 -- 172.21.15.46:0/3997002224 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67880095f0 con 0x7f67940ffb90 2024-04-27T03:58:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.210+0000 7f679259c700 1 --2- 172.21.15.46:0/3997002224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67940ffb90 0x7f6794196710 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f678400b480 tx=0x7f678400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:58:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.210+0000 7f67837fe700 1 -- 172.21.15.46:0/3997002224 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6784011800 con 0x7f67940ffb90 2024-04-27T03:58:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.210+0000 7f67837fe700 1 -- 172.21.15.46:0/3997002224 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6784011e20 con 0x7f67940ffb90 2024-04-27T03:58:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.210+0000 7f6798cdc700 1 -- 172.21.15.46:0/3997002224 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679419be10 con 0x7f67940ffb90 2024-04-27T03:58:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.210+0000 7f67837fe700 1 -- 172.21.15.46:0/3997002224 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6784010ea0 con 0x7f67940ffb90 2024-04-27T03:58:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.210+0000 7f6798cdc700 1 -- 172.21.15.46:0/3997002224 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679419c350 con 0x7f67940ffb90 2024-04-27T03:58:23.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.211+0000 7f6798cdc700 1 -- 172.21.15.46:0/3997002224 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679404ebb0 con 0x7f67940ffb90 2024-04-27T03:58:23.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.212+0000 7f67837fe700 1 -- 172.21.15.46:0/3997002224 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7f6784010380 con 0x7f67940ffb90 2024-04-27T03:58:23.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.212+0000 7f67837fe700 1 --2- 172.21.15.46:0/3997002224 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f677c06cc50 0x7f677c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:58:23.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.212+0000 7f67837fe700 1 -- 172.21.15.46:0/3997002224 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7f678408af90 con 0x7f67940ffb90 2024-04-27T03:58:23.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.212+0000 7f6791d9b700 1 --2- 172.21.15.46:0/3997002224 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f677c06cc50 0x7f677c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:58:23.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.213+0000 7f6791d9b700 1 --2- 172.21.15.46:0/3997002224 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f677c06cc50 0x7f677c06f110 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6788006010 tx=0x7f678800b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:58:23.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.217+0000 7f67837fe700 1 -- 172.21.15.46:0/3997002224 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f67840bec50 con 0x7f67940ffb90 2024-04-27T03:58:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:23.432+0000 7f6798cdc700 1 -- 172.21.15.46:0/3997002224 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6794068d00 con 0x7f677c06cc50 2024-04-27T03:58:24.262 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:24 smithi046 ceph-mon[26899]: pgmap v59: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:24.262 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:24 smithi046 ceph-mon[26899]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:58:24.262 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:24 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T03:58:24.262 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:24 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T03:58:24.262 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:24 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:24.262 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:24 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:24.262 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:24 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:24.262 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:24 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:58:24.263 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:24 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:58:24.263 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:24 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:24.263 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:24 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:24.263 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:24 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:58:24.263 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:24 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:24.263 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:24 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:58:24.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:24 smithi073 ceph-mon[34160]: pgmap v59: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:24.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:24 smithi073 ceph-mon[34160]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:58:24.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:24 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T03:58:24.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:24 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T03:58:24.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:24 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:24.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:24 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:24.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:24 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:24.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:24 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:58:24.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:24 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:58:24.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:24 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:24.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:24 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:24.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:24 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:58:24.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:24 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:24.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:24 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:58:25.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:25 smithi046 ceph-mon[26899]: Detected new or changed devices on smithi046 2024-04-27T03:58:25.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:25 smithi046 ceph-mon[26899]: Adjusting osd_memory_target on smithi046 to 5922M 2024-04-27T03:58:25.365 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:25 smithi073 ceph-mon[34160]: Detected new or changed devices on smithi046 2024-04-27T03:58:25.365 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:25 smithi073 ceph-mon[34160]: Adjusting osd_memory_target on smithi046 to 5922M 2024-04-27T03:58:26.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:26 smithi046 ceph-mon[26899]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:26.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:26 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:26.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:26 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:26.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:26 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2927740153' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "addf9aa9-ba34-4443-b239-6df9f7ef68e4"}]: dispatch 2024-04-27T03:58:26.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:26 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2927740153' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "addf9aa9-ba34-4443-b239-6df9f7ef68e4"}]': finished 2024-04-27T03:58:26.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:26 smithi046 ceph-mon[26899]: osdmap e16: 3 total, 2 up, 3 in 2024-04-27T03:58:26.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:26 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T03:58:26.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:26 smithi073 ceph-mon[34160]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:26.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:26 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:26.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:26 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:26.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:26 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/2927740153' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "addf9aa9-ba34-4443-b239-6df9f7ef68e4"}]: dispatch 2024-04-27T03:58:26.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:26 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/2927740153' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "addf9aa9-ba34-4443-b239-6df9f7ef68e4"}]': finished 2024-04-27T03:58:26.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:26 smithi073 ceph-mon[34160]: osdmap e16: 3 total, 2 up, 3 in 2024-04-27T03:58:26.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:26 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T03:58:27.345 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:27 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2488502967' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T03:58:27.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:27 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/2488502967' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T03:58:28.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:28 smithi046 ceph-mon[26899]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:28.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:28 smithi073 ceph-mon[34160]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:30.358 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:30 smithi046 ceph-mon[26899]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:30.358 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:30 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:30.359 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:30 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:30.359 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:30 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:30.359 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:30 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:58:30.359 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:30 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:30.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:30 smithi073 ceph-mon[34160]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:30.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:30 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:30.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:30 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:30.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:30 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:30.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:30 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:58:30.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:30 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:31.565 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:58:31.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:31 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:58:32.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:32 smithi046 ceph-mon[26899]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:32.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:32 smithi073 ceph-mon[34160]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:34.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:34 smithi046 ceph-mon[26899]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:34.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:34 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T03:58:34.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:34 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:34.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:34 smithi046 ceph-mon[26899]: Deploying daemon osd.2 on smithi046 2024-04-27T03:58:34.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:34 smithi073 ceph-mon[34160]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:34.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:34 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T03:58:34.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:34 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:34.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:34 smithi073 ceph-mon[34160]: Deploying daemon osd.2 on smithi046 2024-04-27T03:58:36.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:36 smithi046 ceph-mon[26899]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:36.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:36 smithi073 ceph-mon[34160]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:38.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:38 smithi046 ceph-mon[26899]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:38.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:38 smithi073 ceph-mon[34160]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:39.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:39.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:39.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:58:39.899 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:39.899 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:39.899 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:58:40.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:40 smithi046 ceph-mon[26899]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:41.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:40 smithi073 ceph-mon[34160]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:42.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:42.490+0000 7f67837fe700 1 -- 172.21.15.46:0/3997002224 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6794068d00 con 0x7f677c06cc50 2024-04-27T03:58:42.493 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 2 on host 'smithi046' 2024-04-27T03:58:42.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:42.492+0000 7f6798cdc700 1 -- 172.21.15.46:0/3997002224 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f677c06cc50 msgr2=0x7f677c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:58:42.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:42.492+0000 7f6798cdc700 1 --2- 172.21.15.46:0/3997002224 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f677c06cc50 0x7f677c06f110 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6788006010 tx=0x7f678800b540 comp rx=0 tx=0).stop 2024-04-27T03:58:42.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:42.492+0000 7f6798cdc700 1 -- 172.21.15.46:0/3997002224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67940ffb90 msgr2=0x7f6794196710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:58:42.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:42.492+0000 7f6798cdc700 1 --2- 172.21.15.46:0/3997002224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67940ffb90 0x7f6794196710 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f678400b480 tx=0x7f678400b790 comp rx=0 tx=0).stop 2024-04-27T03:58:42.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:42.492+0000 7f6798cdc700 1 -- 172.21.15.46:0/3997002224 shutdown_connections 2024-04-27T03:58:42.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:42.492+0000 7f6798cdc700 1 --2- 172.21.15.46:0/3997002224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67940ffb90 0x7f6794196710 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:58:42.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:42.492+0000 7f6798cdc700 1 --2- 172.21.15.46:0/3997002224 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f677c06cc50 0x7f677c06f110 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:58:42.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:42.492+0000 7f6798cdc700 1 --2- 172.21.15.46:0/3997002224 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6794100d90 0x7f6794196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:58:42.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:42.492+0000 7f6798cdc700 1 -- 172.21.15.46:0/3997002224 >> 172.21.15.46:0/3997002224 conn(0x7f67940fb110 msgr2=0x7f67940685f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:58:42.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:42.492+0000 7f6798cdc700 1 -- 172.21.15.46:0/3997002224 shutdown_connections 2024-04-27T03:58:42.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T03:58:42.492+0000 7f6798cdc700 1 -- 172.21.15.46:0/3997002224 wait complete. 2024-04-27T03:58:42.799 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:42 smithi046 ceph-mon[26899]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:42.799 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:42 smithi046 ceph-mon[26899]: from='osd.2 [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T03:58:42.799 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:42 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:42.799 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:42 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:43.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:42 smithi073 ceph-mon[34160]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:43.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:42 smithi073 ceph-mon[34160]: from='osd.2 [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T03:58:43.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:42 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:43.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:42 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:43.160 DEBUG:teuthology.orchestra.run.smithi046:osd.2> sudo journalctl -f -n 0 -u ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@osd.2.service 2024-04-27T03:58:43.163 INFO:tasks.cephadm:Deploying osd.3 on smithi073 with /dev/vg_nvme/lv_4... 2024-04-27T03:58:43.163 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-27T03:58:43.374 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi073/config 2024-04-27T03:58:43.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:43 smithi073 ceph-mon[34160]: from='osd.2 [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-27T03:58:43.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:43 smithi073 ceph-mon[34160]: osdmap e17: 3 total, 2 up, 3 in 2024-04-27T03:58:43.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:43 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T03:58:43.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:43 smithi073 ceph-mon[34160]: from='osd.2 [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-27T03:58:43.873 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:43 smithi046 ceph-mon[26899]: from='osd.2 [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-27T03:58:43.873 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:43 smithi046 ceph-mon[26899]: osdmap e17: 3 total, 2 up, 3 in 2024-04-27T03:58:43.873 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:43 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T03:58:43.873 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:43 smithi046 ceph-mon[26899]: from='osd.2 [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-27T03:58:43.873 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 03:58:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T03:58:43.613+0000 7f7e04235700 -1 osd.2 0 waiting for initial osdmap 2024-04-27T03:58:43.874 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 03:58:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T03:58:43.618+0000 7f7e0082b700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T03:58:44.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:44 smithi073 ceph-mon[34160]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:44.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:44 smithi073 ceph-mon[34160]: from='osd.2 [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-27T03:58:44.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:44 smithi073 ceph-mon[34160]: osdmap e18: 3 total, 2 up, 3 in 2024-04-27T03:58:44.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T03:58:44.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T03:58:44.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T03:58:44.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:44 smithi046 ceph-mon[26899]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:44.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:44 smithi046 ceph-mon[26899]: from='osd.2 [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-27T03:58:44.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:44 smithi046 ceph-mon[26899]: osdmap e18: 3 total, 2 up, 3 in 2024-04-27T03:58:44.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T03:58:44.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T03:58:44.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T03:58:45.273 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-27T03:58:45.273 INFO:teuthology.orchestra.run.smithi073.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-27T03:58:45.273 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10+0 records in 2024-04-27T03:58:45.273 INFO:teuthology.orchestra.run.smithi073.stderr:10+0 records out 2024-04-27T03:58:45.273 INFO:teuthology.orchestra.run.smithi073.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0129024 s, 813 MB/s 2024-04-27T03:58:45.273 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping successful for: 2024-04-27T03:58:45.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:45 smithi073 ceph-mon[34160]: purged_snaps scrub starts 2024-04-27T03:58:45.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:45 smithi073 ceph-mon[34160]: purged_snaps scrub ok 2024-04-27T03:58:45.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:45 smithi073 ceph-mon[34160]: osd.2 [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760] boot 2024-04-27T03:58:45.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:45 smithi073 ceph-mon[34160]: osdmap e19: 3 total, 3 up, 3 in 2024-04-27T03:58:45.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:45 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T03:58:45.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:45 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-27T03:58:45.866 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi073:vg_nvme/lv_4 2024-04-27T03:58:45.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:45 smithi046 ceph-mon[26899]: purged_snaps scrub starts 2024-04-27T03:58:45.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:45 smithi046 ceph-mon[26899]: purged_snaps scrub ok 2024-04-27T03:58:45.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:45 smithi046 ceph-mon[26899]: osd.2 [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760] boot 2024-04-27T03:58:45.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:45 smithi046 ceph-mon[26899]: osdmap e19: 3 total, 3 up, 3 in 2024-04-27T03:58:45.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:45 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T03:58:45.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:45 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-27T03:58:46.071 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi073/config 2024-04-27T03:58:46.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:46 smithi073 ceph-mon[34160]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:46.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:46 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-27T03:58:46.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:46 smithi073 ceph-mon[34160]: osdmap e20: 3 total, 3 up, 3 in 2024-04-27T03:58:46.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:46 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T03:58:46.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:46 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:46.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:46 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:46.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:46 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:58:46.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:46 smithi046 ceph-mon[26899]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-27T03:58:46.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:46 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-27T03:58:46.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:46 smithi046 ceph-mon[26899]: osdmap e20: 3 total, 3 up, 3 in 2024-04-27T03:58:46.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:46 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T03:58:46.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:46 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:46.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:46 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:46.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:46 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:58:47.655 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:47 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-27T03:58:47.655 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:47 smithi046 ceph-mon[26899]: osdmap e21: 3 total, 3 up, 3 in 2024-04-27T03:58:47.656 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 03:58:47 smithi046 sudo[63079]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T03:58:47.656 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 03:58:47 smithi046 sudo[63079]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T03:58:47.656 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 03:58:47 smithi046 sudo[63079]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T03:58:47.656 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 03:58:47 smithi046 sudo[63079]: pam_unix(sudo:session): session closed for user root 2024-04-27T03:58:47.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.701+0000 7fd5a5341700 1 -- 172.21.15.73:0/2115278222 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5a0100d90 msgr2=0x7fd5a01011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:58:47.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.701+0000 7fd5a5341700 1 --2- 172.21.15.73:0/2115278222 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5a0100d90 0x7fd5a01011f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd588009910 tx=0x7fd588009c20 comp rx=0 tx=0).stop 2024-04-27T03:58:47.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.701+0000 7fd5a5341700 1 -- 172.21.15.73:0/2115278222 shutdown_connections 2024-04-27T03:58:47.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.701+0000 7fd5a5341700 1 --2- 172.21.15.73:0/2115278222 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5a0100d90 0x7fd5a01011f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:58:47.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.701+0000 7fd5a5341700 1 --2- 172.21.15.73:0/2115278222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5a00ffb90 0x7fd5a00fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:58:47.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.701+0000 7fd5a5341700 1 -- 172.21.15.73:0/2115278222 >> 172.21.15.73:0/2115278222 conn(0x7fd5a00fb110 msgr2=0x7fd5a00fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:58:47.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.701+0000 7fd5a5341700 1 -- 172.21.15.73:0/2115278222 shutdown_connections 2024-04-27T03:58:47.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.701+0000 7fd5a5341700 1 -- 172.21.15.73:0/2115278222 wait complete. 2024-04-27T03:58:47.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.702+0000 7fd5a5341700 1 Processor -- start 2024-04-27T03:58:47.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.702+0000 7fd5a5341700 1 -- start start 2024-04-27T03:58:47.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.702+0000 7fd5a5341700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5a00ffb90 0x7fd5a01966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:58:47.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.702+0000 7fd5a5341700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5a0100d90 0x7fd5a0196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:58:47.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.702+0000 7fd5a5341700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a0197250 con 0x7fd5a00ffb90 2024-04-27T03:58:47.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.702+0000 7fd5a5341700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a0197390 con 0x7fd5a0100d90 2024-04-27T03:58:47.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.702+0000 7fd59e7fc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5a0100d90 0x7fd5a0196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:58:47.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.702+0000 7fd59e7fc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5a0100d90 0x7fd5a0196c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43434/0 (socket says 172.21.15.73:43434) 2024-04-27T03:58:47.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.702+0000 7fd59e7fc700 1 -- 172.21.15.73:0/2431740027 learned_addr learned my addr 172.21.15.73:0/2431740027 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:58:47.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.702+0000 7fd59effd700 1 --2- 172.21.15.73:0/2431740027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5a00ffb90 0x7fd5a01966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:58:47.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.702+0000 7fd59e7fc700 1 -- 172.21.15.73:0/2431740027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5a00ffb90 msgr2=0x7fd5a01966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:58:47.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.702+0000 7fd59e7fc700 1 --2- 172.21.15.73:0/2431740027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5a00ffb90 0x7fd5a01966f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:58:47.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.702+0000 7fd59e7fc700 1 -- 172.21.15.73:0/2431740027 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5880095f0 con 0x7fd5a0100d90 2024-04-27T03:58:47.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.702+0000 7fd59e7fc700 1 --2- 172.21.15.73:0/2431740027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5a0100d90 0x7fd5a0196c30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd588005b40 tx=0x7fd58800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:58:47.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.703+0000 7fd597fff700 1 -- 172.21.15.73:0/2431740027 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd58801d070 con 0x7fd5a0100d90 2024-04-27T03:58:47.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.703+0000 7fd597fff700 1 -- 172.21.15.73:0/2431740027 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5880050d0 con 0x7fd5a0100d90 2024-04-27T03:58:47.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.703+0000 7fd597fff700 1 -- 172.21.15.73:0/2431740027 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd588022470 con 0x7fd5a0100d90 2024-04-27T03:58:47.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.703+0000 7fd5a5341700 1 -- 172.21.15.73:0/2431740027 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a019bd40 con 0x7fd5a0100d90 2024-04-27T03:58:47.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.703+0000 7fd5a5341700 1 -- 172.21.15.73:0/2431740027 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a019c230 con 0x7fd5a0100d90 2024-04-27T03:58:47.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.704+0000 7fd5a5341700 1 -- 172.21.15.73:0/2431740027 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a004ebb0 con 0x7fd5a0100d90 2024-04-27T03:58:47.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.704+0000 7fd597fff700 1 -- 172.21.15.73:0/2431740027 <== mon.1 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7fd5880037f0 con 0x7fd5a0100d90 2024-04-27T03:58:47.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.704+0000 7fd597fff700 1 --2- 172.21.15.73:0/2431740027 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fd58c06cc50 0x7fd58c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:58:47.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.705+0000 7fd597fff700 1 -- 172.21.15.73:0/2431740027 <== mon.1 v2:172.21.15.73:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3232+0+0 (secure 0 0 0) 0x7fd58808cce0 con 0x7fd5a0100d90 2024-04-27T03:58:47.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.705+0000 7fd59effd700 1 --2- 172.21.15.73:0/2431740027 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fd58c06cc50 0x7fd58c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:58:47.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.705+0000 7fd59effd700 1 --2- 172.21.15.73:0/2431740027 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fd58c06cc50 0x7fd58c06f110 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0100bf0 tx=0x7fd59800b040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:58:47.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.709+0000 7fd597fff700 1 -- 172.21.15.73:0/2431740027 <== mon.1 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fd58805b560 con 0x7fd5a0100d90 2024-04-27T03:58:47.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:47 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-27T03:58:47.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:47 smithi073 ceph-mon[34160]: osdmap e21: 3 total, 3 up, 3 in 2024-04-27T03:58:47.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:47.915+0000 7fd5a5341700 1 -- 172.21.15.73:0/2431740027 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5a0068d80 con 0x7fd58c06cc50 2024-04-27T03:58:47.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 03:58:47 smithi046 sudo[63091]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T03:58:47.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 03:58:47 smithi046 sudo[63091]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T03:58:47.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 03:58:47 smithi046 sudo[63091]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T03:58:47.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 03:58:47 smithi046 sudo[63091]: pam_unix(sudo:session): session closed for user root 2024-04-27T03:58:48.640 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 03:58:48 smithi046 sudo[65087]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T03:58:48.641 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 03:58:48 smithi046 sudo[65087]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T03:58:48.641 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 03:58:48 smithi046 sudo[65087]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T03:58:48.641 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 03:58:48 smithi046 sudo[65087]: pam_unix(sudo:session): session closed for user root 2024-04-27T03:58:48.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 03:58:48 smithi046 sudo[65846]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T03:58:48.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 03:58:48 smithi046 sudo[65846]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T03:58:48.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 03:58:48 smithi046 sudo[65846]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T03:58:48.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 03:58:48 smithi046 sudo[65846]: pam_unix(sudo:session): session closed for user root 2024-04-27T03:58:48.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:48 smithi046 ceph-mon[26899]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-27T03:58:48.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:48 smithi046 ceph-mon[26899]: osdmap e22: 3 total, 3 up, 3 in 2024-04-27T03:58:48.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:48 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:48.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:48 smithi046 ceph-mon[26899]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:58:48.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:48 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T03:58:48.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:48 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T03:58:48.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:48 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:49.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:48 smithi073 ceph-mon[34160]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-27T03:58:49.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:48 smithi073 ceph-mon[34160]: osdmap e22: 3 total, 3 up, 3 in 2024-04-27T03:58:49.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:48 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:49.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:48 smithi073 ceph-mon[34160]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:58:49.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:48 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T03:58:49.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:48 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T03:58:49.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:48 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:49.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 03:58:49 smithi046 sudo[65881]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T03:58:49.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 03:58:49 smithi046 sudo[65881]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T03:58:49.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 03:58:49 smithi046 sudo[65881]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T03:58:49.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 03:58:49 smithi046 sudo[65881]: pam_unix(sudo:session): session closed for user root 2024-04-27T03:58:49.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:58:49.648+0000 7fd597fff700 1 -- 172.21.15.73:0/2431740027 <== mon.1 v2:172.21.15.73:3300/0 7 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd588057ea0 con 0x7fd5a0100d90 2024-04-27T03:58:50.063 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 03:58:49 smithi046 sudo[66014]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T03:58:50.064 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 03:58:49 smithi046 sudo[66014]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T03:58:50.064 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 03:58:49 smithi046 sudo[66014]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T03:58:50.064 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 03:58:49 smithi046 sudo[66014]: pam_unix(sudo:session): session closed for user root 2024-04-27T03:58:50.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 sudo[66056]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T03:58:50.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 sudo[66056]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T03:58:50.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 sudo[66056]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T03:58:50.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 sudo[66056]: pam_unix(sudo:session): session closed for user root 2024-04-27T03:58:50.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 sudo[66123]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T03:58:50.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 sudo[66123]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T03:58:50.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 sudo[66123]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T03:58:50.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 sudo[66123]: pam_unix(sudo:session): session closed for user root 2024-04-27T03:58:50.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 ceph-mon[26899]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-27T03:58:50.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 ceph-mon[26899]: Detected new or changed devices on smithi046 2024-04-27T03:58:50.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:50.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:50.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:58:50.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:58:50.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:58:50.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 ceph-mon[26899]: Adjusting osd_memory_target on smithi046 to 3948M 2024-04-27T03:58:50.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:50.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:50.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:58:50.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:50.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:58:50.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 ceph-mon[26899]: mgrmap e19: smithi046.qrfdvr(active, since 2m), standbys: smithi073.awyllt 2024-04-27T03:58:50.926 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:50 smithi046 ceph-mon[26899]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T03:58:50.928 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 ceph-mon[34160]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-27T03:58:50.928 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 ceph-mon[34160]: Detected new or changed devices on smithi046 2024-04-27T03:58:50.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:50.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:50.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:58:50.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:58:50.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:58:50.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 ceph-mon[34160]: Adjusting osd_memory_target on smithi046 to 3948M 2024-04-27T03:58:50.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:50.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:50.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:58:50.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:50.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:58:50.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 ceph-mon[34160]: mgrmap e19: smithi046.qrfdvr(active, since 2m), standbys: smithi073.awyllt 2024-04-27T03:58:50.930 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 ceph-mon[34160]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T03:58:50.930 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 sudo[38144]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T03:58:50.930 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 sudo[38144]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T03:58:50.930 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 sudo[38144]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T03:58:50.930 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:50 smithi073 sudo[38144]: pam_unix(sudo:session): session closed for user root 2024-04-27T03:58:51.256 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 sudo[38147]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T03:58:51.256 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 sudo[38147]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T03:58:51.256 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 sudo[38147]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T03:58:51.256 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 sudo[38147]: pam_unix(sudo:session): session closed for user root 2024-04-27T03:58:51.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 ceph-mon[34160]: from='client.? 172.21.15.73:0/679160725' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fa6eb8e-8816-4308-894d-575c57e335d7"}]: dispatch 2024-04-27T03:58:51.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 ceph-mon[34160]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T03:58:51.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-04-27T03:58:51.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 ceph-mon[34160]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fa6eb8e-8816-4308-894d-575c57e335d7"}]: dispatch 2024-04-27T03:58:51.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-04-27T03:58:51.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 ceph-mon[34160]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T03:58:51.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 ceph-mon[34160]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fa6eb8e-8816-4308-894d-575c57e335d7"}]': finished 2024-04-27T03:58:51.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 ceph-mon[34160]: osdmap e23: 4 total, 3 up, 4 in 2024-04-27T03:58:51.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T03:58:51.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-04-27T03:58:51.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-04-27T03:58:51.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 ceph-mon[34160]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-27T03:58:51.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 ceph-mon[34160]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T03:58:51.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:51 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:51.878 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:51 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/679160725' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fa6eb8e-8816-4308-894d-575c57e335d7"}]: dispatch 2024-04-27T03:58:51.878 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:51 smithi046 ceph-mon[26899]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T03:58:51.878 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:51 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-04-27T03:58:51.878 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:51 smithi046 ceph-mon[26899]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fa6eb8e-8816-4308-894d-575c57e335d7"}]: dispatch 2024-04-27T03:58:51.878 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:51 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-04-27T03:58:51.878 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:51 smithi046 ceph-mon[26899]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T03:58:51.878 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:51 smithi046 ceph-mon[26899]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fa6eb8e-8816-4308-894d-575c57e335d7"}]': finished 2024-04-27T03:58:51.879 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:51 smithi046 ceph-mon[26899]: osdmap e23: 4 total, 3 up, 4 in 2024-04-27T03:58:51.879 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:51 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T03:58:51.879 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:51 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-04-27T03:58:51.879 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:51 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-04-27T03:58:51.879 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:51 smithi046 ceph-mon[26899]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-27T03:58:51.879 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:51 smithi046 ceph-mon[26899]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T03:58:51.879 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:51 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:52.928 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:52 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/1265676045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T03:58:53.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:52 smithi073 ceph-mon[34160]: from='client.? 172.21.15.73:0/1265676045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T03:58:54.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:54 smithi073 ceph-mon[34160]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-27T03:58:54.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:54 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:54.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:54 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:54.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:54 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:54.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:54 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:58:54.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:54 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:54.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:54 smithi046 ceph-mon[26899]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-27T03:58:54.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:54 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:54.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:54 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:54.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:54 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:54.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:54 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:58:54.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:54 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:58:56.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:56 smithi073 ceph-mon[34160]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-27T03:58:56.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:56 smithi046 ceph-mon[26899]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-27T03:58:58.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:58 smithi073 ceph-mon[34160]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-27T03:58:58.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:58 smithi046 ceph-mon[26899]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-27T03:58:59.765 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:59 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T03:58:59.765 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:59 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:59.765 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:58:59 smithi073 ceph-mon[34160]: Deploying daemon osd.3 on smithi073 2024-04-27T03:58:59.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:59 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T03:58:59.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:59 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:58:59.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:58:59 smithi046 ceph-mon[26899]: Deploying daemon osd.3 on smithi073 2024-04-27T03:59:00.652 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:00 smithi073 ceph-mon[34160]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-27T03:59:00.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:00 smithi046 ceph-mon[26899]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-27T03:59:01.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:01 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:59:01.825 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:01 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:59:02.791 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:02 smithi073 ceph-mon[34160]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-27T03:59:02.836 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:02 smithi046 ceph-mon[26899]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-27T03:59:04.650 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:04 smithi073 ceph-mon[34160]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-27T03:59:04.650 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:04 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:04.650 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:04 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:04.650 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:04 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:59:04.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:04 smithi046 ceph-mon[26899]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-27T03:59:04.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:04 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:04.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:04 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:04.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:04 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:59:06.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:06 smithi073 ceph-mon[34160]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-27T03:59:06.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:06 smithi046 ceph-mon[26899]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-27T03:59:07.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:07.126+0000 7fd597fff700 1 -- 172.21.15.73:0/2431740027 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd5a0068d80 con 0x7fd58c06cc50 2024-04-27T03:59:07.129 INFO:teuthology.orchestra.run.smithi073.stdout:Created osd(s) 3 on host 'smithi073' 2024-04-27T03:59:07.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:07.129+0000 7fd5a5341700 1 -- 172.21.15.73:0/2431740027 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fd58c06cc50 msgr2=0x7fd58c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:59:07.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:07.129+0000 7fd5a5341700 1 --2- 172.21.15.73:0/2431740027 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fd58c06cc50 0x7fd58c06f110 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0100bf0 tx=0x7fd59800b040 comp rx=0 tx=0).stop 2024-04-27T03:59:07.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:07.129+0000 7fd5a5341700 1 -- 172.21.15.73:0/2431740027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5a0100d90 msgr2=0x7fd5a0196c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:59:07.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:07.129+0000 7fd5a5341700 1 --2- 172.21.15.73:0/2431740027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5a0100d90 0x7fd5a0196c30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd588005b40 tx=0x7fd58800f690 comp rx=0 tx=0).stop 2024-04-27T03:59:07.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:07.129+0000 7fd5a5341700 1 -- 172.21.15.73:0/2431740027 shutdown_connections 2024-04-27T03:59:07.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:07.129+0000 7fd5a5341700 1 --2- 172.21.15.73:0/2431740027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5a00ffb90 0x7fd5a01966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:59:07.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:07.129+0000 7fd5a5341700 1 --2- 172.21.15.73:0/2431740027 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fd58c06cc50 0x7fd58c06f110 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:59:07.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:07.129+0000 7fd5a5341700 1 --2- 172.21.15.73:0/2431740027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5a0100d90 0x7fd5a0196c30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:59:07.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:07.129+0000 7fd5a5341700 1 -- 172.21.15.73:0/2431740027 >> 172.21.15.73:0/2431740027 conn(0x7fd5a00fb110 msgr2=0x7fd5a0068670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:59:07.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:07.129+0000 7fd5a5341700 1 -- 172.21.15.73:0/2431740027 shutdown_connections 2024-04-27T03:59:07.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:07.129+0000 7fd5a5341700 1 -- 172.21.15.73:0/2431740027 wait complete. 2024-04-27T03:59:07.902 DEBUG:teuthology.orchestra.run.smithi073:osd.3> sudo journalctl -f -n 0 -u ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@osd.3.service 2024-04-27T03:59:07.904 INFO:tasks.cephadm:Deploying osd.4 on smithi073 with /dev/vg_nvme/lv_3... 2024-04-27T03:59:07.905 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-27T03:59:08.323 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi073/config 2024-04-27T03:59:08.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:08 smithi073 ceph-mon[34160]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-27T03:59:08.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:08 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:08.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:08 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:08.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:08 smithi073 ceph-mon[34160]: from='osd.3 [v2:172.21.15.73:6800/2427344041,v1:172.21.15.73:6801/2427344041]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T03:59:08.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:08 smithi073 ceph-mon[34160]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T03:59:08.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:08 smithi046 ceph-mon[26899]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-27T03:59:08.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:08 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:08.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:08 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:08.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:08 smithi046 ceph-mon[26899]: from='osd.3 [v2:172.21.15.73:6800/2427344041,v1:172.21.15.73:6801/2427344041]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T03:59:08.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:08 smithi046 ceph-mon[26899]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T03:59:09.641 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:09 smithi073 ceph-mon[34160]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-27T03:59:09.641 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:09 smithi073 ceph-mon[34160]: osdmap e24: 4 total, 3 up, 4 in 2024-04-27T03:59:09.641 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:09 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T03:59:09.641 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:09 smithi073 ceph-mon[34160]: from='osd.3 [v2:172.21.15.73:6800/2427344041,v1:172.21.15.73:6801/2427344041]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-04-27T03:59:09.641 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:09 smithi073 ceph-mon[34160]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-04-27T03:59:09.641 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:09 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:09.641 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:09 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:09.641 INFO:journalctl@ceph.osd.3.smithi073.stdout:Apr 27 03:59:09 smithi073 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-3[40221]: 2024-04-27T03:59:09.538+0000 7f1e96089700 -1 osd.3 0 waiting for initial osdmap 2024-04-27T03:59:09.641 INFO:journalctl@ceph.osd.3.smithi073.stdout:Apr 27 03:59:09 smithi073 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-3[40221]: 2024-04-27T03:59:09.544+0000 7f1e9267f700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T03:59:09.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:09 smithi046 ceph-mon[26899]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-27T03:59:09.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:09 smithi046 ceph-mon[26899]: osdmap e24: 4 total, 3 up, 4 in 2024-04-27T03:59:09.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:09 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T03:59:09.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:09 smithi046 ceph-mon[26899]: from='osd.3 [v2:172.21.15.73:6800/2427344041,v1:172.21.15.73:6801/2427344041]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-04-27T03:59:09.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:09 smithi046 ceph-mon[26899]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-04-27T03:59:09.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:09 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:09.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:09 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:10.264 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-27T03:59:10.265 INFO:teuthology.orchestra.run.smithi073.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-27T03:59:10.265 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10+0 records in 2024-04-27T03:59:10.265 INFO:teuthology.orchestra.run.smithi073.stderr:10+0 records out 2024-04-27T03:59:10.265 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128272 s, 817 MB/s 2024-04-27T03:59:10.265 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping successful for: 2024-04-27T03:59:10.798 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:10 smithi073 ceph-mon[34160]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-27T03:59:10.799 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:10 smithi073 ceph-mon[34160]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-04-27T03:59:10.799 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:10 smithi073 ceph-mon[34160]: osdmap e25: 4 total, 3 up, 4 in 2024-04-27T03:59:10.799 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:10 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T03:59:10.799 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:10 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T03:59:10.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:10 smithi046 ceph-mon[26899]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-27T03:59:10.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:10 smithi046 ceph-mon[26899]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-04-27T03:59:10.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:10 smithi046 ceph-mon[26899]: osdmap e25: 4 total, 3 up, 4 in 2024-04-27T03:59:10.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:10 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T03:59:10.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:10 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T03:59:11.115 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi073:vg_nvme/lv_3 2024-04-27T03:59:11.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:11 smithi073 ceph-mon[34160]: purged_snaps scrub starts 2024-04-27T03:59:11.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:11 smithi073 ceph-mon[34160]: purged_snaps scrub ok 2024-04-27T03:59:11.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:11 smithi073 ceph-mon[34160]: osd.3 [v2:172.21.15.73:6800/2427344041,v1:172.21.15.73:6801/2427344041] boot 2024-04-27T03:59:11.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:11 smithi073 ceph-mon[34160]: osdmap e26: 4 total, 4 up, 4 in 2024-04-27T03:59:11.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:11 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T03:59:11.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:11 smithi046 ceph-mon[26899]: purged_snaps scrub starts 2024-04-27T03:59:11.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:11 smithi046 ceph-mon[26899]: purged_snaps scrub ok 2024-04-27T03:59:11.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:11 smithi046 ceph-mon[26899]: osd.3 [v2:172.21.15.73:6800/2427344041,v1:172.21.15.73:6801/2427344041] boot 2024-04-27T03:59:11.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:11 smithi046 ceph-mon[26899]: osdmap e26: 4 total, 4 up, 4 in 2024-04-27T03:59:11.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:11 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T03:59:12.075 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi073/config 2024-04-27T03:59:12.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:12 smithi073 ceph-mon[34160]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-27T03:59:12.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:12 smithi073 ceph-mon[34160]: osdmap e27: 4 total, 4 up, 4 in 2024-04-27T03:59:12.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:12 smithi046 ceph-mon[26899]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-27T03:59:12.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:12 smithi046 ceph-mon[26899]: osdmap e27: 4 total, 4 up, 4 in 2024-04-27T03:59:13.772 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:13 smithi073 ceph-mon[34160]: osdmap e28: 4 total, 4 up, 4 in 2024-04-27T03:59:13.772 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:13 smithi073 ceph-mon[34160]: osdmap e29: 4 total, 4 up, 4 in 2024-04-27T03:59:13.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:13 smithi046 ceph-mon[26899]: osdmap e28: 4 total, 4 up, 4 in 2024-04-27T03:59:13.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:13 smithi046 ceph-mon[26899]: osdmap e29: 4 total, 4 up, 4 in 2024-04-27T03:59:14.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.152+0000 7fbdf39f8700 1 -- 172.21.15.73:0/3971841095 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbdec100d40 msgr2=0x7fbdec1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:59:14.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.152+0000 7fbdf39f8700 1 --2- 172.21.15.73:0/3971841095 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbdec100d40 0x7fbdec1011a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbddc009910 tx=0x7fbddc009c20 comp rx=0 tx=0).stop 2024-04-27T03:59:14.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.152+0000 7fbdf39f8700 1 -- 172.21.15.73:0/3971841095 shutdown_connections 2024-04-27T03:59:14.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.152+0000 7fbdf39f8700 1 --2- 172.21.15.73:0/3971841095 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbdec100d40 0x7fbdec1011a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:59:14.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.152+0000 7fbdf39f8700 1 --2- 172.21.15.73:0/3971841095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdec0ffb40 0x7fbdec0fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:59:14.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.152+0000 7fbdf39f8700 1 -- 172.21.15.73:0/3971841095 >> 172.21.15.73:0/3971841095 conn(0x7fbdec0fb0e0 msgr2=0x7fbdec0fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:59:14.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.152+0000 7fbdf39f8700 1 -- 172.21.15.73:0/3971841095 shutdown_connections 2024-04-27T03:59:14.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.152+0000 7fbdf39f8700 1 -- 172.21.15.73:0/3971841095 wait complete. 2024-04-27T03:59:14.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.153+0000 7fbdf39f8700 1 Processor -- start 2024-04-27T03:59:14.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.153+0000 7fbdf39f8700 1 -- start start 2024-04-27T03:59:14.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.153+0000 7fbdf39f8700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbdec0ffb40 0x7fbdec196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:59:14.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.153+0000 7fbdf39f8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdec100d40 0x7fbdec196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:59:14.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.153+0000 7fbdf39f8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec1971b0 con 0x7fbdec100d40 2024-04-27T03:59:14.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.153+0000 7fbdf39f8700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec1972f0 con 0x7fbdec0ffb40 2024-04-27T03:59:14.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.153+0000 7fbdf1794700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbdec0ffb40 0x7fbdec196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:59:14.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.153+0000 7fbdf1794700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbdec0ffb40 0x7fbdec196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49342/0 (socket says 172.21.15.73:49342) 2024-04-27T03:59:14.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.153+0000 7fbdf1794700 1 -- 172.21.15.73:0/2111599272 learned_addr learned my addr 172.21.15.73:0/2111599272 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:59:14.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.153+0000 7fbdf0f93700 1 --2- 172.21.15.73:0/2111599272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdec100d40 0x7fbdec196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:59:14.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.154+0000 7fbdf1794700 1 -- 172.21.15.73:0/2111599272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdec100d40 msgr2=0x7fbdec196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:59:14.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.154+0000 7fbdf1794700 1 --2- 172.21.15.73:0/2111599272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdec100d40 0x7fbdec196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:59:14.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.154+0000 7fbdf1794700 1 -- 172.21.15.73:0/2111599272 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbddc0095f0 con 0x7fbdec0ffb40 2024-04-27T03:59:14.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.154+0000 7fbdf0f93700 1 --2- 172.21.15.73:0/2111599272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdec100d40 0x7fbdec196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T03:59:14.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.154+0000 7fbdf1794700 1 --2- 172.21.15.73:0/2111599272 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbdec0ffb40 0x7fbdec196650 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbde800d900 tx=0x7fbde800dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:59:14.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.154+0000 7fbde27fc700 1 -- 172.21.15.73:0/2111599272 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbde80041f0 con 0x7fbdec0ffb40 2024-04-27T03:59:14.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.154+0000 7fbde27fc700 1 -- 172.21.15.73:0/2111599272 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbde8004350 con 0x7fbdec0ffb40 2024-04-27T03:59:14.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.154+0000 7fbdf39f8700 1 -- 172.21.15.73:0/2111599272 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdec19bd50 con 0x7fbdec0ffb40 2024-04-27T03:59:14.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.154+0000 7fbde27fc700 1 -- 172.21.15.73:0/2111599272 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbde8003cf0 con 0x7fbdec0ffb40 2024-04-27T03:59:14.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.154+0000 7fbdf39f8700 1 -- 172.21.15.73:0/2111599272 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdec19c2c0 con 0x7fbdec0ffb40 2024-04-27T03:59:14.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.155+0000 7fbdf39f8700 1 -- 172.21.15.73:0/2111599272 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdec04ec20 con 0x7fbdec0ffb40 2024-04-27T03:59:14.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.155+0000 7fbde27fc700 1 -- 172.21.15.73:0/2111599272 <== mon.1 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbde8009730 con 0x7fbdec0ffb40 2024-04-27T03:59:14.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.156+0000 7fbde27fc700 1 --2- 172.21.15.73:0/2111599272 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbdd806cce0 0x7fbdd806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:59:14.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.156+0000 7fbde27fc700 1 -- 172.21.15.73:0/2111599272 <== mon.1 v2:172.21.15.73:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3774+0+0 (secure 0 0 0) 0x7fbde808b980 con 0x7fbdec0ffb40 2024-04-27T03:59:14.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.156+0000 7fbdf0f93700 1 --2- 172.21.15.73:0/2111599272 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbdd806cce0 0x7fbdd806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:59:14.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.157+0000 7fbdf0f93700 1 --2- 172.21.15.73:0/2111599272 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbdd806cce0 0x7fbdd806f1a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbddc005090 tx=0x7fbddc005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:59:14.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.161+0000 7fbde27fc700 1 -- 172.21.15.73:0/2111599272 <== mon.1 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fbdec04ec20 con 0x7fbdec0ffb40 2024-04-27T03:59:14.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:14.360+0000 7fbdf39f8700 1 -- 172.21.15.73:0/2111599272 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdec04ebb0 con 0x7fbdd806cce0 2024-04-27T03:59:14.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:14 smithi073 ceph-mon[34160]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:14.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:14.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:14.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:59:14.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:14.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:59:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:59:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:59:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:59:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:59:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T03:59:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T03:59:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:14 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:59:14.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:14 smithi073 ceph-mon[34160]: osdmap e30: 4 total, 4 up, 4 in 2024-04-27T03:59:14.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:14 smithi046 ceph-mon[26899]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:14.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:14.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:14.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:59:14.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:14.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:59:14.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:59:14.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:14.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:59:14.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:59:14.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:59:14.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:14.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T03:59:14.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T03:59:14.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:14 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:59:14.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:14 smithi046 ceph-mon[26899]: osdmap e30: 4 total, 4 up, 4 in 2024-04-27T03:59:15.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:15 smithi073 ceph-mon[34160]: Detected new or changed devices on smithi073 2024-04-27T03:59:15.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:15 smithi073 ceph-mon[34160]: Adjusting osd_memory_target on smithi073 to 14912M 2024-04-27T03:59:15.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:15 smithi073 ceph-mon[34160]: from='client.24165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:59:15.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:15 smithi046 ceph-mon[26899]: Detected new or changed devices on smithi073 2024-04-27T03:59:15.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:15 smithi046 ceph-mon[26899]: Adjusting osd_memory_target on smithi073 to 14912M 2024-04-27T03:59:15.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:15 smithi046 ceph-mon[26899]: from='client.24165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:59:16.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:16 smithi073 ceph-mon[34160]: pgmap v100: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-27T03:59:16.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:16 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:59:16.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:16 smithi046 ceph-mon[26899]: pgmap v100: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-27T03:59:16.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:16 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:59:17.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:17 smithi073 ceph-mon[34160]: from='client.? 172.21.15.73:0/4107459388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb0a02c1-cc1a-4c2a-a469-0c941b9fe3fb"}]: dispatch 2024-04-27T03:59:17.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:17 smithi073 ceph-mon[34160]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb0a02c1-cc1a-4c2a-a469-0c941b9fe3fb"}]: dispatch 2024-04-27T03:59:17.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:17 smithi073 ceph-mon[34160]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb0a02c1-cc1a-4c2a-a469-0c941b9fe3fb"}]': finished 2024-04-27T03:59:17.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:17 smithi073 ceph-mon[34160]: osdmap e31: 5 total, 4 up, 5 in 2024-04-27T03:59:17.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:17 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T03:59:17.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:17 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/4107459388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb0a02c1-cc1a-4c2a-a469-0c941b9fe3fb"}]: dispatch 2024-04-27T03:59:17.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:17 smithi046 ceph-mon[26899]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb0a02c1-cc1a-4c2a-a469-0c941b9fe3fb"}]: dispatch 2024-04-27T03:59:17.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:17 smithi046 ceph-mon[26899]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb0a02c1-cc1a-4c2a-a469-0c941b9fe3fb"}]': finished 2024-04-27T03:59:17.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:17 smithi046 ceph-mon[26899]: osdmap e31: 5 total, 4 up, 5 in 2024-04-27T03:59:17.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:17 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T03:59:18.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:18 smithi073 ceph-mon[34160]: pgmap v101: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-27T03:59:18.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:18 smithi073 ceph-mon[34160]: from='client.? 172.21.15.73:0/2847945563' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T03:59:18.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:18 smithi046 ceph-mon[26899]: pgmap v101: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-27T03:59:18.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:18 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/2847945563' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T03:59:20.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:20 smithi073 ceph-mon[34160]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-27T03:59:20.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:20 smithi046 ceph-mon[26899]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-27T03:59:22.852 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:22 smithi073 ceph-mon[34160]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:22.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:22 smithi046 ceph-mon[26899]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:24.920 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:24 smithi073 ceph-mon[34160]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:24.920 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:24 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T03:59:24.920 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:24 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:59:24.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:24 smithi046 ceph-mon[26899]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:24.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:24 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T03:59:24.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:24 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:59:25.850 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:25 smithi073 ceph-mon[34160]: Deploying daemon osd.4 on smithi073 2024-04-27T03:59:25.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:25 smithi046 ceph-mon[26899]: Deploying daemon osd.4 on smithi073 2024-04-27T03:59:26.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:26 smithi073 ceph-mon[34160]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:26.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:26 smithi046 ceph-mon[26899]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:28.896 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:28 smithi073 ceph-mon[34160]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:28.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:28 smithi046 ceph-mon[26899]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:30.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:30 smithi073 ceph-mon[34160]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:30.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:30 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:30.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:30 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:30.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:30 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:59:30.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:30 smithi046 ceph-mon[26899]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:30.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:30 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:30.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:30 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:30.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:30 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:59:31.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:31 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:59:31.824 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:59:32.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:32.475+0000 7fbde27fc700 1 -- 172.21.15.73:0/2111599272 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbdec04ebb0 con 0x7fbdd806cce0 2024-04-27T03:59:32.478 INFO:teuthology.orchestra.run.smithi073.stdout:Created osd(s) 4 on host 'smithi073' 2024-04-27T03:59:32.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:32.478+0000 7fbdf39f8700 1 -- 172.21.15.73:0/2111599272 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbdd806cce0 msgr2=0x7fbdd806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:59:32.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:32.478+0000 7fbdf39f8700 1 --2- 172.21.15.73:0/2111599272 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbdd806cce0 0x7fbdd806f1a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbddc005090 tx=0x7fbddc005960 comp rx=0 tx=0).stop 2024-04-27T03:59:32.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:32.478+0000 7fbdf39f8700 1 -- 172.21.15.73:0/2111599272 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbdec0ffb40 msgr2=0x7fbdec196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:59:32.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:32.478+0000 7fbdf39f8700 1 --2- 172.21.15.73:0/2111599272 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbdec0ffb40 0x7fbdec196650 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbde800d900 tx=0x7fbde800dc10 comp rx=0 tx=0).stop 2024-04-27T03:59:32.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:32.478+0000 7fbdf39f8700 1 -- 172.21.15.73:0/2111599272 shutdown_connections 2024-04-27T03:59:32.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:32.478+0000 7fbdf39f8700 1 --2- 172.21.15.73:0/2111599272 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbdd806cce0 0x7fbdd806f1a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:59:32.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:32.478+0000 7fbdf39f8700 1 --2- 172.21.15.73:0/2111599272 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbdec0ffb40 0x7fbdec196650 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:59:32.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:32.478+0000 7fbdf39f8700 1 --2- 172.21.15.73:0/2111599272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdec100d40 0x7fbdec196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:59:32.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:32.478+0000 7fbdf39f8700 1 -- 172.21.15.73:0/2111599272 >> 172.21.15.73:0/2111599272 conn(0x7fbdec0fb0e0 msgr2=0x7fbdec0691a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:59:32.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:32.478+0000 7fbdf39f8700 1 -- 172.21.15.73:0/2111599272 shutdown_connections 2024-04-27T03:59:32.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:32.478+0000 7fbdf39f8700 1 -- 172.21.15.73:0/2111599272 wait complete. 2024-04-27T03:59:32.821 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:32 smithi073 ceph-mon[34160]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:32.821 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:32 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:32.821 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:32 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:32.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:32 smithi046 ceph-mon[26899]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:32.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:32 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:32.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:32 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:33.450 DEBUG:teuthology.orchestra.run.smithi073:osd.4> sudo journalctl -f -n 0 -u ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@osd.4.service 2024-04-27T03:59:33.452 INFO:tasks.cephadm:Deploying osd.5 on smithi073 with /dev/vg_nvme/lv_2... 2024-04-27T03:59:33.453 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-27T03:59:33.746 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:33 smithi073 ceph-mon[34160]: from='osd.4 [v2:172.21.15.73:6808/1194795747,v1:172.21.15.73:6809/1194795747]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T03:59:33.747 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:33 smithi073 ceph-mon[34160]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T03:59:33.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:33 smithi046 ceph-mon[26899]: from='osd.4 [v2:172.21.15.73:6808/1194795747,v1:172.21.15.73:6809/1194795747]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T03:59:33.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:33 smithi046 ceph-mon[26899]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T03:59:34.331 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi073/config 2024-04-27T03:59:34.776 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:34 smithi073 ceph-mon[34160]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:34.776 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:34 smithi073 ceph-mon[34160]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-27T03:59:34.776 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:34 smithi073 ceph-mon[34160]: osdmap e32: 5 total, 4 up, 5 in 2024-04-27T03:59:34.776 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:34 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T03:59:34.776 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:34 smithi073 ceph-mon[34160]: from='osd.4 [v2:172.21.15.73:6808/1194795747,v1:172.21.15.73:6809/1194795747]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-04-27T03:59:34.776 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:34 smithi073 ceph-mon[34160]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-04-27T03:59:34.776 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:34 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:34.776 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:34 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:34.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:34 smithi046 ceph-mon[26899]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:34.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:34 smithi046 ceph-mon[26899]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-27T03:59:34.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:34 smithi046 ceph-mon[26899]: osdmap e32: 5 total, 4 up, 5 in 2024-04-27T03:59:34.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:34 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T03:59:34.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:34 smithi046 ceph-mon[26899]: from='osd.4 [v2:172.21.15.73:6808/1194795747,v1:172.21.15.73:6809/1194795747]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-04-27T03:59:34.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:34 smithi046 ceph-mon[26899]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-04-27T03:59:34.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:34 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:34.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:34 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:35.370 INFO:journalctl@ceph.osd.4.smithi073.stdout:Apr 27 03:59:35 smithi073 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-4[47107]: 2024-04-27T03:59:35.326+0000 7fa7ad417700 -1 osd.4 0 waiting for initial osdmap 2024-04-27T03:59:35.371 INFO:journalctl@ceph.osd.4.smithi073.stdout:Apr 27 03:59:35 smithi073 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-4[47107]: 2024-04-27T03:59:35.334+0000 7fa7a5a05700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T03:59:35.657 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:35 smithi073 ceph-mon[34160]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-04-27T03:59:35.658 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:35 smithi073 ceph-mon[34160]: osdmap e33: 5 total, 4 up, 5 in 2024-04-27T03:59:35.658 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:35 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T03:59:35.658 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:35 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T03:59:35.658 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:35 smithi073 ceph-mon[34160]: from='osd.4 ' entity='osd.4' 2024-04-27T03:59:35.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:35 smithi046 ceph-mon[26899]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-04-27T03:59:35.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:35 smithi046 ceph-mon[26899]: osdmap e33: 5 total, 4 up, 5 in 2024-04-27T03:59:35.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:35 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T03:59:35.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:35 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T03:59:35.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:35 smithi046 ceph-mon[26899]: from='osd.4 ' entity='osd.4' 2024-04-27T03:59:36.253 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-27T03:59:36.253 INFO:teuthology.orchestra.run.smithi073.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-27T03:59:36.253 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10+0 records in 2024-04-27T03:59:36.253 INFO:teuthology.orchestra.run.smithi073.stderr:10+0 records out 2024-04-27T03:59:36.253 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0137122 s, 765 MB/s 2024-04-27T03:59:36.253 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping successful for: 2024-04-27T03:59:36.518 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:36 smithi073 ceph-mon[34160]: purged_snaps scrub starts 2024-04-27T03:59:36.518 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:36 smithi073 ceph-mon[34160]: purged_snaps scrub ok 2024-04-27T03:59:36.519 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:36 smithi073 ceph-mon[34160]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:36.519 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:36 smithi073 ceph-mon[34160]: osd.4 [v2:172.21.15.73:6808/1194795747,v1:172.21.15.73:6809/1194795747] boot 2024-04-27T03:59:36.519 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:36 smithi073 ceph-mon[34160]: osdmap e34: 5 total, 5 up, 5 in 2024-04-27T03:59:36.519 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:36 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T03:59:36.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:36 smithi046 ceph-mon[26899]: purged_snaps scrub starts 2024-04-27T03:59:36.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:36 smithi046 ceph-mon[26899]: purged_snaps scrub ok 2024-04-27T03:59:36.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:36 smithi046 ceph-mon[26899]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-27T03:59:36.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:36 smithi046 ceph-mon[26899]: osd.4 [v2:172.21.15.73:6808/1194795747,v1:172.21.15.73:6809/1194795747] boot 2024-04-27T03:59:36.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:36 smithi046 ceph-mon[26899]: osdmap e34: 5 total, 5 up, 5 in 2024-04-27T03:59:36.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:36 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T03:59:37.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:37 smithi073 ceph-mon[34160]: osdmap e35: 5 total, 5 up, 5 in 2024-04-27T03:59:37.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:37 smithi046 ceph-mon[26899]: osdmap e35: 5 total, 5 up, 5 in 2024-04-27T03:59:38.100 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi073:vg_nvme/lv_2 2024-04-27T03:59:38.499 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi073/config 2024-04-27T03:59:38.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:38 smithi046 ceph-mon[26899]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 547 MiB used, 446 GiB / 447 GiB avail 2024-04-27T03:59:38.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:38 smithi073 ceph-mon[34160]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 547 MiB used, 446 GiB / 447 GiB avail 2024-04-27T03:59:40.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:39 smithi046 ceph-mon[26899]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 547 MiB used, 446 GiB / 447 GiB avail 2024-04-27T03:59:40.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:39 smithi046 ceph-mon[26899]: Detected new or changed devices on smithi073 2024-04-27T03:59:40.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:40.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:40.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:59:40.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:59:40.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:39 smithi046 ceph-mon[26899]: Adjusting osd_memory_target on smithi073 to 7456M 2024-04-27T03:59:40.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:40.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:59:40.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:59:40.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:40.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:59:40.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:59:40.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:59:40.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:40.176 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:39 smithi073 ceph-mon[34160]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 547 MiB used, 446 GiB / 447 GiB avail 2024-04-27T03:59:40.176 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:39 smithi073 ceph-mon[34160]: Detected new or changed devices on smithi073 2024-04-27T03:59:40.176 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:40.176 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:40.176 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:59:40.176 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T03:59:40.176 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:39 smithi073 ceph-mon[34160]: Adjusting osd_memory_target on smithi073 to 7456M 2024-04-27T03:59:40.176 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:40.176 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:59:40.177 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:59:40.177 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:40.177 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:59:40.177 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:59:40.177 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T03:59:40.177 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:40.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.367+0000 7f39bdcc7700 1 -- 172.21.15.73:0/681933955 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39b8101e10 msgr2=0x7f39b8102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:59:40.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.367+0000 7f39bdcc7700 1 --2- 172.21.15.73:0/681933955 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39b8101e10 0x7f39b8102270 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f39ac009910 tx=0x7f39ac009c20 comp rx=0 tx=0).stop 2024-04-27T03:59:40.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.367+0000 7f39bdcc7700 1 -- 172.21.15.73:0/681933955 shutdown_connections 2024-04-27T03:59:40.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.367+0000 7f39bdcc7700 1 --2- 172.21.15.73:0/681933955 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39b8101e10 0x7f39b8102270 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:59:40.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.367+0000 7f39bdcc7700 1 --2- 172.21.15.73:0/681933955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39b8100c10 0x7f39b8101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:59:40.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.367+0000 7f39bdcc7700 1 -- 172.21.15.73:0/681933955 >> 172.21.15.73:0/681933955 conn(0x7f39b80fc1f0 msgr2=0x7f39b80fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:59:40.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.367+0000 7f39bdcc7700 1 -- 172.21.15.73:0/681933955 shutdown_connections 2024-04-27T03:59:40.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.367+0000 7f39bdcc7700 1 -- 172.21.15.73:0/681933955 wait complete. 2024-04-27T03:59:40.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.368+0000 7f39bdcc7700 1 Processor -- start 2024-04-27T03:59:40.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.368+0000 7f39bdcc7700 1 -- start start 2024-04-27T03:59:40.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.368+0000 7f39bdcc7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39b8100c10 0x7f39b8196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:59:40.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.368+0000 7f39bdcc7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39b8101e10 0x7f39b8196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:59:40.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.368+0000 7f39bdcc7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b8197190 con 0x7f39b8100c10 2024-04-27T03:59:40.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.368+0000 7f39bdcc7700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b81972d0 con 0x7f39b8101e10 2024-04-27T03:59:40.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.368+0000 7f39b6ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39b8101e10 0x7f39b8196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:59:40.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.368+0000 7f39b6ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39b8101e10 0x7f39b8196b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40472/0 (socket says 172.21.15.73:40472) 2024-04-27T03:59:40.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.368+0000 7f39b6ffd700 1 -- 172.21.15.73:0/809451948 learned_addr learned my addr 172.21.15.73:0/809451948 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T03:59:40.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.368+0000 7f39b77fe700 1 --2- 172.21.15.73:0/809451948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39b8100c10 0x7f39b8196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:59:40.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.368+0000 7f39b6ffd700 1 -- 172.21.15.73:0/809451948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39b8100c10 msgr2=0x7f39b8196630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:59:40.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.368+0000 7f39b6ffd700 1 --2- 172.21.15.73:0/809451948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39b8100c10 0x7f39b8196630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:59:40.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.368+0000 7f39b6ffd700 1 -- 172.21.15.73:0/809451948 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac0095f0 con 0x7f39b8101e10 2024-04-27T03:59:40.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.369+0000 7f39b6ffd700 1 --2- 172.21.15.73:0/809451948 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39b8101e10 0x7f39b8196b70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f39ac006010 tx=0x7f39ac00bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:59:40.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.369+0000 7f39b4ff9700 1 -- 172.21.15.73:0/809451948 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39ac01d070 con 0x7f39b8101e10 2024-04-27T03:59:40.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.369+0000 7f39bdcc7700 1 -- 172.21.15.73:0/809451948 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b819bc80 con 0x7f39b8101e10 2024-04-27T03:59:40.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.369+0000 7f39b4ff9700 1 -- 172.21.15.73:0/809451948 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39ac005dc0 con 0x7f39b8101e10 2024-04-27T03:59:40.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.369+0000 7f39b4ff9700 1 -- 172.21.15.73:0/809451948 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39ac021410 con 0x7f39b8101e10 2024-04-27T03:59:40.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.369+0000 7f39bdcc7700 1 -- 172.21.15.73:0/809451948 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b819c1f0 con 0x7f39b8101e10 2024-04-27T03:59:40.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.370+0000 7f39bdcc7700 1 -- 172.21.15.73:0/809451948 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b8190790 con 0x7f39b8101e10 2024-04-27T03:59:40.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.370+0000 7f39b4ff9700 1 -- 172.21.15.73:0/809451948 <== mon.1 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f39ac00f460 con 0x7f39b8101e10 2024-04-27T03:59:40.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.371+0000 7f39b4ff9700 1 --2- 172.21.15.73:0/809451948 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f39a006ccf0 0x7f39a006f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:59:40.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.371+0000 7f39b4ff9700 1 -- 172.21.15.73:0/809451948 <== mon.1 v2:172.21.15.73:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4206+0+0 (secure 0 0 0) 0x7f39ac08cf90 con 0x7f39b8101e10 2024-04-27T03:59:40.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.371+0000 7f39b77fe700 1 --2- 172.21.15.73:0/809451948 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f39a006ccf0 0x7f39a006f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:59:40.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.371+0000 7f39b77fe700 1 --2- 172.21.15.73:0/809451948 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f39a006ccf0 0x7f39a006f1b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f39b8101c70 tx=0x7f39a800b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:59:40.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.375+0000 7f39b4ff9700 1 -- 172.21.15.73:0/809451948 <== mon.1 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f39ac05b2e0 con 0x7f39b8101e10 2024-04-27T03:59:40.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:40.580+0000 7f39bdcc7700 1 -- 172.21.15.73:0/809451948 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f39b802cea0 con 0x7f39a006ccf0 2024-04-27T03:59:41.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:40 smithi046 ceph-mon[26899]: from='client.24183 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:59:41.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:40 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T03:59:41.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:40 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T03:59:41.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:40 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:59:41.181 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:40 smithi073 ceph-mon[34160]: from='client.24183 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T03:59:41.181 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:40 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T03:59:41.181 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:40 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T03:59:41.181 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:40 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:59:42.014 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:41 smithi073 ceph-mon[34160]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:42.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:41 smithi046 ceph-mon[26899]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:44.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:43 smithi073 ceph-mon[34160]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:44.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:43 smithi073 ceph-mon[34160]: from='client.? 172.21.15.73:0/1949824677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9d4c4d4-5633-4e9a-ba64-460134d132ac"}]: dispatch 2024-04-27T03:59:44.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:43 smithi073 ceph-mon[34160]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9d4c4d4-5633-4e9a-ba64-460134d132ac"}]: dispatch 2024-04-27T03:59:44.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:43 smithi073 ceph-mon[34160]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9d4c4d4-5633-4e9a-ba64-460134d132ac"}]': finished 2024-04-27T03:59:44.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:43 smithi073 ceph-mon[34160]: osdmap e36: 6 total, 5 up, 6 in 2024-04-27T03:59:44.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:43 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T03:59:44.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:43 smithi073 ceph-mon[34160]: from='client.? 172.21.15.73:0/970375146' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T03:59:44.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:43 smithi046 ceph-mon[26899]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:44.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:43 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/1949824677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9d4c4d4-5633-4e9a-ba64-460134d132ac"}]: dispatch 2024-04-27T03:59:44.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:43 smithi046 ceph-mon[26899]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9d4c4d4-5633-4e9a-ba64-460134d132ac"}]: dispatch 2024-04-27T03:59:44.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:43 smithi046 ceph-mon[26899]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9d4c4d4-5633-4e9a-ba64-460134d132ac"}]': finished 2024-04-27T03:59:44.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:43 smithi046 ceph-mon[26899]: osdmap e36: 6 total, 5 up, 6 in 2024-04-27T03:59:44.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:43 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T03:59:44.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:43 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/970375146' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T03:59:46.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:45 smithi046 ceph-mon[26899]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:46.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:45 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:59:46.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:45 smithi073 ceph-mon[34160]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:46.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:45 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T03:59:48.173 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:47 smithi073 ceph-mon[34160]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:48.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:47 smithi046 ceph-mon[26899]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:50.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:49 smithi073 ceph-mon[34160]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:50.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:49 smithi046 ceph-mon[26899]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:50.977 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:50 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T03:59:50.977 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:50 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:59:50.977 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:50 smithi073 ceph-mon[34160]: Deploying daemon osd.5 on smithi073 2024-04-27T03:59:51.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T03:59:51.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:50 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T03:59:51.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:50 smithi046 ceph-mon[26899]: Deploying daemon osd.5 on smithi073 2024-04-27T03:59:52.002 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:51 smithi073 ceph-mon[34160]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:52.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:51 smithi046 ceph-mon[26899]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:54.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:53 smithi046 ceph-mon[26899]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:54.238 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:53 smithi073 ceph-mon[34160]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:56.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:55 smithi073 ceph-mon[34160]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:56.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:55 smithi046 ceph-mon[26899]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:57.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:56 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:57.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:56 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:57.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:56 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:59:57.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:56 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:57.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:56 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T03:59:57.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:56 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T03:59:58.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 03:59:57 smithi073 ceph-mon[34160]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:58.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 03:59:57 smithi046 ceph-mon[26899]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T03:59:59.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:59.145+0000 7f39b4ff9700 1 -- 172.21.15.73:0/809451948 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f39b802cea0 con 0x7f39a006ccf0 2024-04-27T03:59:59.147 INFO:teuthology.orchestra.run.smithi073.stdout:Created osd(s) 5 on host 'smithi073' 2024-04-27T03:59:59.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:59.148+0000 7f39bdcc7700 1 -- 172.21.15.73:0/809451948 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f39a006ccf0 msgr2=0x7f39a006f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:59:59.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:59.148+0000 7f39bdcc7700 1 --2- 172.21.15.73:0/809451948 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f39a006ccf0 0x7f39a006f1b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f39b8101c70 tx=0x7f39a800b410 comp rx=0 tx=0).stop 2024-04-27T03:59:59.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:59.148+0000 7f39bdcc7700 1 -- 172.21.15.73:0/809451948 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39b8101e10 msgr2=0x7f39b8196b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:59:59.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:59.148+0000 7f39bdcc7700 1 --2- 172.21.15.73:0/809451948 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39b8101e10 0x7f39b8196b70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f39ac006010 tx=0x7f39ac00bd70 comp rx=0 tx=0).stop 2024-04-27T03:59:59.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:59.148+0000 7f39bdcc7700 1 -- 172.21.15.73:0/809451948 shutdown_connections 2024-04-27T03:59:59.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:59.148+0000 7f39bdcc7700 1 --2- 172.21.15.73:0/809451948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39b8100c10 0x7f39b8196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:59:59.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:59.148+0000 7f39bdcc7700 1 --2- 172.21.15.73:0/809451948 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f39a006ccf0 0x7f39a006f1b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:59:59.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:59.148+0000 7f39bdcc7700 1 --2- 172.21.15.73:0/809451948 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39b8101e10 0x7f39b8196b70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:59:59.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:59.148+0000 7f39bdcc7700 1 -- 172.21.15.73:0/809451948 >> 172.21.15.73:0/809451948 conn(0x7f39b80fc1f0 msgr2=0x7f39b8105410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:59:59.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:59.148+0000 7f39bdcc7700 1 -- 172.21.15.73:0/809451948 shutdown_connections 2024-04-27T03:59:59.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T03:59:59.148+0000 7f39bdcc7700 1 -- 172.21.15.73:0/809451948 wait complete. 2024-04-27T03:59:59.814 DEBUG:teuthology.orchestra.run.smithi073:osd.5> sudo journalctl -f -n 0 -u ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@osd.5.service 2024-04-27T03:59:59.816 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-27T03:59:59.816 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-27T04:00:00.041 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:00.076 INFO:journalctl@ceph.osd.5.smithi073.stdout:Apr 27 03:59:59 smithi073 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-5[54102]: 2024-04-27T03:59:59.887+0000 7f8d7039b7c0 -1 osd.5 0 log_to_monitors true 2024-04-27T04:00:00.381 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:00 smithi046 ceph-mon[26899]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T04:00:00.382 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:00 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:00.382 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:00 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:00.382 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:00 smithi046 ceph-mon[26899]: from='osd.5 [v2:172.21.15.73:6816/4277072929,v1:172.21.15.73:6817/4277072929]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T04:00:00.382 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:00 smithi046 ceph-mon[26899]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T04:00:00.382 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:00 smithi046 ceph-mon[26899]: overall HEALTH_OK 2024-04-27T04:00:00.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:00 smithi073 ceph-mon[34160]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T04:00:00.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:00 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:00.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:00 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:00.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:00 smithi073 ceph-mon[34160]: from='osd.5 [v2:172.21.15.73:6816/4277072929,v1:172.21.15.73:6817/4277072929]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T04:00:00.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:00 smithi073 ceph-mon[34160]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T04:00:00.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:00 smithi073 ceph-mon[34160]: overall HEALTH_OK 2024-04-27T04:00:01.398 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:01 smithi046 ceph-mon[26899]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-27T04:00:01.399 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:01 smithi046 ceph-mon[26899]: osdmap e37: 6 total, 5 up, 6 in 2024-04-27T04:00:01.399 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:01 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T04:00:01.399 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:01 smithi046 ceph-mon[26899]: from='osd.5 [v2:172.21.15.73:6816/4277072929,v1:172.21.15.73:6817/4277072929]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-04-27T04:00:01.399 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:01 smithi046 ceph-mon[26899]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-04-27T04:00:01.399 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:01 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:00:01.399 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:01 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:01.399 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:01 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:01.531 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:01 smithi073 ceph-mon[34160]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-27T04:00:01.531 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:01 smithi073 ceph-mon[34160]: osdmap e37: 6 total, 5 up, 6 in 2024-04-27T04:00:01.531 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:01 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T04:00:01.531 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:01 smithi073 ceph-mon[34160]: from='osd.5 [v2:172.21.15.73:6816/4277072929,v1:172.21.15.73:6817/4277072929]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-04-27T04:00:01.531 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:01 smithi073 ceph-mon[34160]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-04-27T04:00:01.531 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:01 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:00:01.531 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:01 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:01.531 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:01 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.755+0000 7fb2d42eb700 1 -- 172.21.15.46:0/1373852992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2cc1040f0 msgr2=0x7fb2cc1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.755+0000 7fb2d42eb700 1 --2- 172.21.15.46:0/1373852992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2cc1040f0 0x7fb2cc1064e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0009910 tx=0x7fb2c0009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.755+0000 7fb2d42eb700 1 -- 172.21.15.46:0/1373852992 shutdown_connections 2024-04-27T04:00:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.755+0000 7fb2d42eb700 1 --2- 172.21.15.46:0/1373852992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2cc1040f0 0x7fb2cc1064e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.755+0000 7fb2d42eb700 1 --2- 172.21.15.46:0/1373852992 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2cc1017c0 0x7fb2cc103bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.755+0000 7fb2d42eb700 1 -- 172.21.15.46:0/1373852992 >> 172.21.15.46:0/1373852992 conn(0x7fb2cc0fb110 msgr2=0x7fb2cc0fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.755+0000 7fb2d42eb700 1 -- 172.21.15.46:0/1373852992 shutdown_connections 2024-04-27T04:00:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.755+0000 7fb2d42eb700 1 -- 172.21.15.46:0/1373852992 wait complete. 2024-04-27T04:00:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.756+0000 7fb2d42eb700 1 Processor -- start 2024-04-27T04:00:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.756+0000 7fb2d42eb700 1 -- start start 2024-04-27T04:00:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.756+0000 7fb2d42eb700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2cc1017c0 0x7fb2cc196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.756+0000 7fb2d42eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2cc1040f0 0x7fb2cc196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.756+0000 7fb2d42eb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc1971b0 con 0x7fb2cc1040f0 2024-04-27T04:00:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.756+0000 7fb2d42eb700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc1972f0 con 0x7fb2cc1017c0 2024-04-27T04:00:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.756+0000 7fb2d1886700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2cc1040f0 0x7fb2cc196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.756+0000 7fb2d2087700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2cc1017c0 0x7fb2cc196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.756+0000 7fb2d1886700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2cc1040f0 0x7fb2cc196b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42430/0 (socket says 172.21.15.46:42430) 2024-04-27T04:00:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.756+0000 7fb2d1886700 1 -- 172.21.15.46:0/766891895 learned_addr learned my addr 172.21.15.46:0/766891895 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.756+0000 7fb2d2087700 1 -- 172.21.15.46:0/766891895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2cc1040f0 msgr2=0x7fb2cc196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.756+0000 7fb2d2087700 1 --2- 172.21.15.46:0/766891895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2cc1040f0 0x7fb2cc196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.756+0000 7fb2d2087700 1 -- 172.21.15.46:0/766891895 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c00095f0 con 0x7fb2cc1017c0 2024-04-27T04:00:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.757+0000 7fb2d2087700 1 --2- 172.21.15.46:0/766891895 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2cc1017c0 0x7fb2cc196650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00efc0 tx=0x7fb2bc00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.757+0000 7fb2cb7fe700 1 -- 172.21.15.46:0/766891895 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2bc018070 con 0x7fb2cc1017c0 2024-04-27T04:00:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.757+0000 7fb2d42eb700 1 -- 172.21.15.46:0/766891895 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2cc19bcf0 con 0x7fb2cc1017c0 2024-04-27T04:00:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.757+0000 7fb2cb7fe700 1 -- 172.21.15.46:0/766891895 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb2bc00f040 con 0x7fb2cc1017c0 2024-04-27T04:00:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.757+0000 7fb2cb7fe700 1 -- 172.21.15.46:0/766891895 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2bc004020 con 0x7fb2cc1017c0 2024-04-27T04:00:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.757+0000 7fb2d42eb700 1 -- 172.21.15.46:0/766891895 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2cc19c1e0 con 0x7fb2cc1017c0 2024-04-27T04:00:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.758+0000 7fb2d42eb700 1 -- 172.21.15.46:0/766891895 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2cc04ebb0 con 0x7fb2cc1017c0 2024-04-27T04:00:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.758+0000 7fb2cb7fe700 1 -- 172.21.15.46:0/766891895 <== mon.1 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb2bc004ad0 con 0x7fb2cc1017c0 2024-04-27T04:00:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.759+0000 7fb2cb7fe700 1 --2- 172.21.15.46:0/766891895 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb2b406cce0 0x7fb2b406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.759+0000 7fb2cb7fe700 1 -- 172.21.15.46:0/766891895 <== mon.1 v2:172.21.15.73:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4358+0+0 (secure 0 0 0) 0x7fb2bc08bdb0 con 0x7fb2cc1017c0 2024-04-27T04:00:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.759+0000 7fb2d1886700 1 --2- 172.21.15.46:0/766891895 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb2b406cce0 0x7fb2b406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.759+0000 7fb2d1886700 1 --2- 172.21.15.46:0/766891895 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb2b406cce0 0x7fb2b406f1a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0009fd0 tx=0x7fb2c0005940 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:01.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.764+0000 7fb2cb7fe700 1 -- 172.21.15.46:0/766891895 <== mon.1 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fb2bc059fc0 con 0x7fb2cc1017c0 2024-04-27T04:00:01.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.967+0000 7fb2d42eb700 1 -- 172.21.15.46:0/766891895 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fb2cc109820 con 0x7fb2cc1017c0 2024-04-27T04:00:01.969 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:00:01.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.967+0000 7fb2cb7fe700 1 -- 172.21.15.46:0/766891895 <== mon.1 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7fb2bc059b50 con 0x7fb2cc1017c0 2024-04-27T04:00:01.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.969+0000 7fb2d42eb700 1 -- 172.21.15.46:0/766891895 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb2b406cce0 msgr2=0x7fb2b406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:01.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.969+0000 7fb2d42eb700 1 --2- 172.21.15.46:0/766891895 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb2b406cce0 0x7fb2b406f1a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0009fd0 tx=0x7fb2c0005940 comp rx=0 tx=0).stop 2024-04-27T04:00:01.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.969+0000 7fb2d42eb700 1 -- 172.21.15.46:0/766891895 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2cc1017c0 msgr2=0x7fb2cc196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:01.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.969+0000 7fb2d42eb700 1 --2- 172.21.15.46:0/766891895 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2cc1017c0 0x7fb2cc196650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00efc0 tx=0x7fb2bc00c3a0 comp rx=0 tx=0).stop 2024-04-27T04:00:01.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.969+0000 7fb2d42eb700 1 -- 172.21.15.46:0/766891895 shutdown_connections 2024-04-27T04:00:01.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.969+0000 7fb2d42eb700 1 --2- 172.21.15.46:0/766891895 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb2b406cce0 0x7fb2b406f1a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:01.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.969+0000 7fb2d42eb700 1 --2- 172.21.15.46:0/766891895 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2cc1017c0 0x7fb2cc196650 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:01.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.969+0000 7fb2d42eb700 1 --2- 172.21.15.46:0/766891895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2cc1040f0 0x7fb2cc196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:01.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.969+0000 7fb2d42eb700 1 -- 172.21.15.46:0/766891895 >> 172.21.15.46:0/766891895 conn(0x7fb2cc0fb110 msgr2=0x7fb2cc0fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:01.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.969+0000 7fb2d42eb700 1 -- 172.21.15.46:0/766891895 shutdown_connections 2024-04-27T04:00:01.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:01.970+0000 7fb2d42eb700 1 -- 172.21.15.46:0/766891895 wait complete. 2024-04-27T04:00:02.131 INFO:journalctl@ceph.osd.5.smithi073.stdout:Apr 27 04:00:01 smithi073 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-5[54102]: 2024-04-27T04:00:01.969+0000 7f8d651ff700 -1 osd.5 0 waiting for initial osdmap 2024-04-27T04:00:02.131 INFO:journalctl@ceph.osd.5.smithi073.stdout:Apr 27 04:00:01 smithi073 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-5[54102]: 2024-04-27T04:00:01.979+0000 7f8d5d7ed700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T04:00:02.273 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:02 smithi046 ceph-mon[26899]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T04:00:02.273 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:02 smithi046 ceph-mon[26899]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-04-27T04:00:02.273 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:02 smithi046 ceph-mon[26899]: osdmap e38: 6 total, 5 up, 6 in 2024-04-27T04:00:02.274 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:02 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T04:00:02.274 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:02 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T04:00:02.274 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:02 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/766891895' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-27T04:00:02.274 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:02 smithi046 ceph-mon[26899]: from='osd.5 ' entity='osd.5' 2024-04-27T04:00:02.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:02 smithi073 ceph-mon[34160]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T04:00:02.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:02 smithi073 ceph-mon[34160]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-04-27T04:00:02.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:02 smithi073 ceph-mon[34160]: osdmap e38: 6 total, 5 up, 6 in 2024-04-27T04:00:02.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:02 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T04:00:02.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:02 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T04:00:02.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:02 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/766891895' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-27T04:00:02.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:02 smithi073 ceph-mon[34160]: from='osd.5 ' entity='osd.5' 2024-04-27T04:00:02.706 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1714190375,"num_in_osds":6,"osd_in_since":1714190383,"num_remapped_pgs":0} 2024-04-27T04:00:03.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:03 smithi046 ceph-mon[26899]: purged_snaps scrub starts 2024-04-27T04:00:03.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:03 smithi046 ceph-mon[26899]: purged_snaps scrub ok 2024-04-27T04:00:03.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T04:00:03.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:03 smithi046 ceph-mon[26899]: osd.5 [v2:172.21.15.73:6816/4277072929,v1:172.21.15.73:6817/4277072929] boot 2024-04-27T04:00:03.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:03 smithi046 ceph-mon[26899]: osdmap e39: 6 total, 6 up, 6 in 2024-04-27T04:00:03.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T04:00:03.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:03 smithi073 ceph-mon[34160]: purged_snaps scrub starts 2024-04-27T04:00:03.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:03 smithi073 ceph-mon[34160]: purged_snaps scrub ok 2024-04-27T04:00:03.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:03 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T04:00:03.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:03 smithi073 ceph-mon[34160]: osd.5 [v2:172.21.15.73:6816/4277072929,v1:172.21.15.73:6817/4277072929] boot 2024-04-27T04:00:03.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:03 smithi073 ceph-mon[34160]: osdmap e39: 6 total, 6 up, 6 in 2024-04-27T04:00:03.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:03 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T04:00:03.707 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-27T04:00:03.922 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:04.242 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:04 smithi073 ceph-mon[34160]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T04:00:04.351 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:04 smithi046 ceph-mon[26899]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-27T04:00:05.315 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:05 smithi046 ceph-mon[26899]: osdmap e40: 6 total, 6 up, 6 in 2024-04-27T04:00:05.315 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:05 smithi046 ceph-mon[26899]: Detected new or changed devices on smithi073 2024-04-27T04:00:05.315 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:05 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:05.315 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:05 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:05.315 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:05 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T04:00:05.315 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:05 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T04:00:05.316 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:05 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T04:00:05.316 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:05 smithi046 ceph-mon[26899]: Adjusting osd_memory_target on smithi073 to 4970M 2024-04-27T04:00:05.316 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:05 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:05.316 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:05 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:00:05.316 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:05 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:00:05.316 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:05 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:05.316 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:05 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:00:05.316 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:05 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:00:05.316 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:05 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:00:05.316 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:05 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:05.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:05 smithi073 ceph-mon[34160]: osdmap e40: 6 total, 6 up, 6 in 2024-04-27T04:00:05.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:05 smithi073 ceph-mon[34160]: Detected new or changed devices on smithi073 2024-04-27T04:00:05.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:05 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:05.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:05 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:05.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:05 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T04:00:05.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:05 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T04:00:05.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:05 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T04:00:05.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:05 smithi073 ceph-mon[34160]: Adjusting osd_memory_target on smithi073 to 4970M 2024-04-27T04:00:05.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:05 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:05.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:05 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:00:05.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:05 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:00:05.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:05 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:05.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:05 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:00:05.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:05 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:00:05.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:05 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:00:05.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:05 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:00:05.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.627+0000 7f2efbaf5700 1 -- 172.21.15.46:0/3952854539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef41040f0 msgr2=0x7f2ef41064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:05.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.627+0000 7f2efbaf5700 1 --2- 172.21.15.46:0/3952854539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef41040f0 0x7f2ef41064e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4009910 tx=0x7f2ee4009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:05.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.627+0000 7f2efbaf5700 1 -- 172.21.15.46:0/3952854539 shutdown_connections 2024-04-27T04:00:05.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.627+0000 7f2efbaf5700 1 --2- 172.21.15.46:0/3952854539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef41040f0 0x7f2ef41064e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:05.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.627+0000 7f2efbaf5700 1 --2- 172.21.15.46:0/3952854539 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ef41017c0 0x7f2ef4103bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:05.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.627+0000 7f2efbaf5700 1 -- 172.21.15.46:0/3952854539 >> 172.21.15.46:0/3952854539 conn(0x7f2ef40fb110 msgr2=0x7f2ef40fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:05.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.627+0000 7f2efbaf5700 1 -- 172.21.15.46:0/3952854539 shutdown_connections 2024-04-27T04:00:05.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.627+0000 7f2efbaf5700 1 -- 172.21.15.46:0/3952854539 wait complete. 2024-04-27T04:00:05.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.628+0000 7f2efbaf5700 1 Processor -- start 2024-04-27T04:00:05.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.628+0000 7f2efbaf5700 1 -- start start 2024-04-27T04:00:05.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.628+0000 7f2efbaf5700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ef41017c0 0x7f2ef41010d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:05.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.628+0000 7f2efbaf5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef41040f0 0x7f2ef40ff770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:05.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.628+0000 7f2efbaf5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40ffcb0 con 0x7f2ef41040f0 2024-04-27T04:00:05.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.628+0000 7f2efbaf5700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40ffdf0 con 0x7f2ef41017c0 2024-04-27T04:00:05.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.628+0000 7f2ef9090700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef41040f0 0x7f2ef40ff770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:05.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.628+0000 7f2ef9090700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef41040f0 0x7f2ef40ff770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42452/0 (socket says 172.21.15.46:42452) 2024-04-27T04:00:05.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.628+0000 7f2ef9090700 1 -- 172.21.15.46:0/3230061882 learned_addr learned my addr 172.21.15.46:0/3230061882 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:05.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.628+0000 7f2ef9090700 1 -- 172.21.15.46:0/3230061882 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ef41017c0 msgr2=0x7f2ef41010d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:05.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.628+0000 7f2ef9891700 1 --2- 172.21.15.46:0/3230061882 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ef41017c0 0x7f2ef41010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:05.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.628+0000 7f2ef9090700 1 --2- 172.21.15.46:0/3230061882 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ef41017c0 0x7f2ef41010d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:05.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.628+0000 7f2ef9090700 1 -- 172.21.15.46:0/3230061882 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee40095f0 con 0x7f2ef41040f0 2024-04-27T04:00:05.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.628+0000 7f2ef9891700 1 --2- 172.21.15.46:0/3230061882 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ef41017c0 0x7f2ef41010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T04:00:05.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.629+0000 7f2ef9090700 1 --2- 172.21.15.46:0/3230061882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef41040f0 0x7f2ef40ff770 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4006010 tx=0x7f2ee400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:05.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.629+0000 7f2eeaffd700 1 -- 172.21.15.46:0/3230061882 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ee401d070 con 0x7f2ef41040f0 2024-04-27T04:00:05.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.629+0000 7f2eeaffd700 1 -- 172.21.15.46:0/3230061882 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ee400f460 con 0x7f2ef41040f0 2024-04-27T04:00:05.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.629+0000 7f2eeaffd700 1 -- 172.21.15.46:0/3230061882 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ee4021640 con 0x7f2ef41040f0 2024-04-27T04:00:05.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.629+0000 7f2efbaf5700 1 -- 172.21.15.46:0/3230061882 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef4100070 con 0x7f2ef41040f0 2024-04-27T04:00:05.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.629+0000 7f2efbaf5700 1 -- 172.21.15.46:0/3230061882 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef4100510 con 0x7f2ef41040f0 2024-04-27T04:00:05.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.630+0000 7f2efbaf5700 1 -- 172.21.15.46:0/3230061882 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef404ebb0 con 0x7f2ef41040f0 2024-04-27T04:00:05.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.630+0000 7f2eeaffd700 1 -- 172.21.15.46:0/3230061882 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2ee400fad0 con 0x7f2ef41040f0 2024-04-27T04:00:05.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.631+0000 7f2eeaffd700 1 --2- 172.21.15.46:0/3230061882 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2ee006ccf0 0x7f2ee006f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:05.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.631+0000 7f2eeaffd700 1 -- 172.21.15.46:0/3230061882 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2ee408d280 con 0x7f2ef41040f0 2024-04-27T04:00:05.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.631+0000 7f2ef9891700 1 --2- 172.21.15.46:0/3230061882 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2ee006ccf0 0x7f2ee006f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:05.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.631+0000 7f2ef9891700 1 --2- 172.21.15.46:0/3230061882 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2ee006ccf0 0x7f2ee006f1b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0007900 tx=0x7f2ef0008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:05.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.636+0000 7f2eeaffd700 1 -- 172.21.15.46:0/3230061882 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f2ee405b420 con 0x7f2ef41040f0 2024-04-27T04:00:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.838+0000 7f2efbaf5700 1 -- 172.21.15.46:0/3230061882 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f2ef419e8d0 con 0x7f2ef41040f0 2024-04-27T04:00:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.838+0000 7f2eeaffd700 1 -- 172.21.15.46:0/3230061882 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f2ee405afb0 con 0x7f2ef41040f0 2024-04-27T04:00:05.840 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:00:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.840+0000 7f2efbaf5700 1 -- 172.21.15.46:0/3230061882 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2ee006ccf0 msgr2=0x7f2ee006f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.840+0000 7f2efbaf5700 1 --2- 172.21.15.46:0/3230061882 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2ee006ccf0 0x7f2ee006f1b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0007900 tx=0x7f2ef0008040 comp rx=0 tx=0).stop 2024-04-27T04:00:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.840+0000 7f2efbaf5700 1 -- 172.21.15.46:0/3230061882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef41040f0 msgr2=0x7f2ef40ff770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.840+0000 7f2efbaf5700 1 --2- 172.21.15.46:0/3230061882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef41040f0 0x7f2ef40ff770 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4006010 tx=0x7f2ee400be20 comp rx=0 tx=0).stop 2024-04-27T04:00:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.840+0000 7f2efbaf5700 1 -- 172.21.15.46:0/3230061882 shutdown_connections 2024-04-27T04:00:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.840+0000 7f2efbaf5700 1 --2- 172.21.15.46:0/3230061882 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2ee006ccf0 0x7f2ee006f1b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.840+0000 7f2efbaf5700 1 --2- 172.21.15.46:0/3230061882 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ef41017c0 0x7f2ef41010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.840+0000 7f2efbaf5700 1 --2- 172.21.15.46:0/3230061882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef41040f0 0x7f2ef40ff770 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.840+0000 7f2efbaf5700 1 -- 172.21.15.46:0/3230061882 >> 172.21.15.46:0/3230061882 conn(0x7f2ef40fb110 msgr2=0x7f2ef40fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.840+0000 7f2efbaf5700 1 -- 172.21.15.46:0/3230061882 shutdown_connections 2024-04-27T04:00:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:05.840+0000 7f2efbaf5700 1 -- 172.21.15.46:0/3230061882 wait complete. 2024-04-27T04:00:06.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:06 smithi046 ceph-mon[26899]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:06.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:06 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/3230061882' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-27T04:00:06.497 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1714190402,"num_in_osds":6,"osd_in_since":1714190383,"num_remapped_pgs":0} 2024-04-27T04:00:06.498 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-27T04:00:06.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:06 smithi073 ceph-mon[34160]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:06.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:06 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/3230061882' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-27T04:00:06.709 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:08.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.283+0000 7f59dc740700 1 -- 172.21.15.46:0/2954397068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59d40ff4a0 msgr2=0x7f59d40ff880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:08.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.283+0000 7f59dc740700 1 --2- 172.21.15.46:0/2954397068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59d40ff4a0 0x7f59d40ff880 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009910 tx=0x7f59d0009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:08.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.283+0000 7f59dc740700 1 -- 172.21.15.46:0/2954397068 shutdown_connections 2024-04-27T04:00:08.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.283+0000 7f59dc740700 1 --2- 172.21.15.46:0/2954397068 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f59d40ffe50 0x7f59d4103ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:08.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.283+0000 7f59dc740700 1 --2- 172.21.15.46:0/2954397068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59d40ff4a0 0x7f59d40ff880 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:08.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.283+0000 7f59dc740700 1 -- 172.21.15.46:0/2954397068 >> 172.21.15.46:0/2954397068 conn(0x7f59d4075f30 msgr2=0x7f59d40fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:08.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.283+0000 7f59dc740700 1 -- 172.21.15.46:0/2954397068 shutdown_connections 2024-04-27T04:00:08.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.283+0000 7f59dc740700 1 -- 172.21.15.46:0/2954397068 wait complete. 2024-04-27T04:00:08.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.284+0000 7f59dc740700 1 Processor -- start 2024-04-27T04:00:08.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.284+0000 7f59dc740700 1 -- start start 2024-04-27T04:00:08.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.284+0000 7f59dc740700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59d40ff4a0 0x7f59d419b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:08.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.284+0000 7f59dc740700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f59d40ffe50 0x7f59d419bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:08.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.284+0000 7f59dc740700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d419c3c0 con 0x7f59d40ff4a0 2024-04-27T04:00:08.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.284+0000 7f59dc740700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d4195930 con 0x7f59d40ffe50 2024-04-27T04:00:08.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.284+0000 7f59da4dc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59d40ff4a0 0x7f59d419b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:08.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.284+0000 7f59da4dc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59d40ff4a0 0x7f59d419b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42468/0 (socket says 172.21.15.46:42468) 2024-04-27T04:00:08.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.284+0000 7f59da4dc700 1 -- 172.21.15.46:0/887681511 learned_addr learned my addr 172.21.15.46:0/887681511 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:08.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.284+0000 7f59d9cdb700 1 --2- 172.21.15.46:0/887681511 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f59d40ffe50 0x7f59d419bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:08.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.284+0000 7f59da4dc700 1 -- 172.21.15.46:0/887681511 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f59d40ffe50 msgr2=0x7f59d419bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:08.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.284+0000 7f59da4dc700 1 --2- 172.21.15.46:0/887681511 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f59d40ffe50 0x7f59d419bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:08.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.284+0000 7f59da4dc700 1 -- 172.21.15.46:0/887681511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d00095f0 con 0x7f59d40ff4a0 2024-04-27T04:00:08.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.284+0000 7f59d9cdb700 1 --2- 172.21.15.46:0/887681511 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f59d40ffe50 0x7f59d419bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T04:00:08.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.284+0000 7f59da4dc700 1 --2- 172.21.15.46:0/887681511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59d40ff4a0 0x7f59d419b7f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f59d000bd70 tx=0x7f59d000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:08.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.285+0000 7f59cb7fe700 1 -- 172.21.15.46:0/887681511 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59d001d070 con 0x7f59d40ff4a0 2024-04-27T04:00:08.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.285+0000 7f59dc740700 1 -- 172.21.15.46:0/887681511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d4195bb0 con 0x7f59d40ff4a0 2024-04-27T04:00:08.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.285+0000 7f59cb7fe700 1 -- 172.21.15.46:0/887681511 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59d000f460 con 0x7f59d40ff4a0 2024-04-27T04:00:08.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.285+0000 7f59cb7fe700 1 -- 172.21.15.46:0/887681511 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59d0017630 con 0x7f59d40ff4a0 2024-04-27T04:00:08.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.285+0000 7f59dc740700 1 -- 172.21.15.46:0/887681511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d41960a0 con 0x7f59d40ff4a0 2024-04-27T04:00:08.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.286+0000 7f59dc740700 1 -- 172.21.15.46:0/887681511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d4100980 con 0x7f59d40ff4a0 2024-04-27T04:00:08.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.286+0000 7f59cb7fe700 1 -- 172.21.15.46:0/887681511 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f59d000f5d0 con 0x7f59d40ff4a0 2024-04-27T04:00:08.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.287+0000 7f59cb7fe700 1 --2- 172.21.15.46:0/887681511 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f59c006cce0 0x7f59c006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:08.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.287+0000 7f59cb7fe700 1 -- 172.21.15.46:0/887681511 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f59d008d490 con 0x7f59d40ff4a0 2024-04-27T04:00:08.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.287+0000 7f59d9cdb700 1 --2- 172.21.15.46:0/887681511 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f59c006cce0 0x7f59c006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:08.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.287+0000 7f59d9cdb700 1 --2- 172.21.15.46:0/887681511 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f59c006cce0 0x7f59c006f1a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f59d4103940 tx=0x7f59c4008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:08.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.291+0000 7f59cb7fe700 1 -- 172.21.15.46:0/887681511 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f59d005b630 con 0x7f59d40ff4a0 2024-04-27T04:00:08.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:08 smithi046 ceph-mon[26899]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:08.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.496+0000 7f59dc740700 1 -- 172.21.15.46:0/887681511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59d410c4e0 con 0x7f59d40ff4a0 2024-04-27T04:00:08.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.497+0000 7f59cb7fe700 1 -- 172.21.15.46:0/887681511 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11301 (secure 0 0 0) 0x7f59d005b1c0 con 0x7f59d40ff4a0 2024-04-27T04:00:08.500 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:00:08.500 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":40,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","created":"2024-04-27T03:54:08.502636+0000","modified":"2024-04-27T04:00:04.148052+0000","last_up_change":"2024-04-27T04:00:02.972167+0000","last_in_change":"2024-04-27T03:59:43.096341+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T03:58:45.241461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0310053-0895-4608-9021-6c04075a1f9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2222489853},{"type":"v1","addr":"172.21.15.46:6803","nonce":2222489853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2222489853},{"type":"v1","addr":"172.21.15.46:6805","nonce":2222489853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2222489853},{"type":"v1","addr":"172.21.15.46:6809","nonce":2222489853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2222489853},{"type":"v1","addr":"172.21.15.46:6807","nonce":2222489853}]},"public_addr":"172.21.15.46:6803/2222489853","cluster_addr":"172.21.15.46:6805/2222489853","heartbeat_back_addr":"172.21.15.46:6809/2222489853","heartbeat_front_addr":"172.21.15.46:6807/2222489853","state":["exists","up"]},{"osd":1,"uuid":"57d1cc25-d5db-41fd-aa72-e380cedf2225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":1807987418},{"type":"v1","addr":"172.21.15.46:6811","nonce":1807987418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":1807987418},{"type":"v1","addr":"172.21.15.46:6813","nonce":1807987418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":1807987418},{"type":"v1","addr":"172.21.15.46:6817","nonce":1807987418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":1807987418},{"type":"v1","addr":"172.21.15.46:6815","nonce":1807987418}]},"public_addr":"172.21.15.46:6811/1807987418","cluster_addr":"172.21.15.46:6813/1807987418","heartbeat_back_addr":"172.21.15.46:6817/1807987418","heartbeat_front_addr":"172.21.15.46:6815/1807987418","state":["exists","up"]},{"osd":2,"uuid":"addf9aa9-ba34-4443-b239-6df9f7ef68e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":933743760},{"type":"v1","addr":"172.21.15.46:6819","nonce":933743760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":933743760},{"type":"v1","addr":"172.21.15.46:6821","nonce":933743760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":933743760},{"type":"v1","addr":"172.21.15.46:6825","nonce":933743760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":933743760},{"type":"v1","addr":"172.21.15.46:6823","nonce":933743760}]},"public_addr":"172.21.15.46:6819/933743760","cluster_addr":"172.21.15.46:6821/933743760","heartbeat_back_addr":"172.21.15.46:6825/933743760","heartbeat_front_addr":"172.21.15.46:6823/933743760","state":["exists","up"]},{"osd":3,"uuid":"4fa6eb8e-8816-4308-894d-575c57e335d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":2427344041},{"type":"v1","addr":"172.21.15.73:6801","nonce":2427344041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":2427344041},{"type":"v1","addr":"172.21.15.73:6803","nonce":2427344041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":2427344041},{"type":"v1","addr":"172.21.15.73:6807","nonce":2427344041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":2427344041},{"type":"v1","addr":"172.21.15.73:6805","nonce":2427344041}]},"public_addr":"172.21.15.73:6801/2427344041","cluster_addr":"172.21.15.73:6803/2427344041","heartbeat_back_addr":"172.21.15.73:6807/2427344041","heartbeat_front_addr":"172.21.15.73:6805/2427344041","state":["exists","up"]},{"osd":4,"uuid":"fb0a02c1-cc1a-4c2a-a469-0c941b9fe3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":1194795747},{"type":"v1","addr":"172.21.15.73:6809","nonce":1194795747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":1194795747},{"type":"v1","addr":"172.21.15.73:6811","nonce":1194795747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":1194795747},{"type":"v1","addr":"172.21.15.73:6815","nonce":1194795747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":1194795747},{"type":"v1","addr":"172.21.15.73:6813","nonce":1194795747}]},"public_addr":"172.21.15.73:6809/1194795747","cluster_addr":"172.21.15.73:6811/1194795747","heartbeat_back_addr":"172.21.15.73:6815/1194795747","heartbeat_front_addr":"172.21.15.73:6813/1194795747","state":["exists","up"]},{"osd":5,"uuid":"b9d4c4d4-5633-4e9a-ba64-460134d132ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":4277072929},{"type":"v1","addr":"172.21.15.73:6817","nonce":4277072929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":4277072929},{"type":"v1","addr":"172.21.15.73:6819","nonce":4277072929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":4277072929},{"type":"v1","addr":"172.21.15.73:6823","nonce":4277072929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":4277072929},{"type":"v1","addr":"172.21.15.73:6821","nonce":4277072929}]},"public_addr":"172.21.15.73:6817/4277072929","cluster_addr":"172.21.15.73:6819/4277072929","heartbeat_back_addr":"172.21.15.73:6823/4277072929","heartbeat_front_addr":"172.21.15.73:6821/4277072929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:57:47.645830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:58:15.796052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:58:43.469886+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:59:08.833298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:59:34.410852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T04:00:00.927552+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6800/3280787672":"2024-04-28T03:56:30.716510+0000","172.21.15.46:0/1728399059":"2024-04-28T03:56:30.716510+0000","172.21.15.46:6801/3280787672":"2024-04-28T03:56:30.716510+0000","172.21.15.46:0/2377183342":"2024-04-28T03:55:14.071598+0000","172.21.15.46:0/1318513098":"2024-04-28T03:54:40.659687+0000","172.21.15.46:0/2660836044":"2024-04-28T03:54:40.659687+0000","172.21.15.46:6801/333567692":"2024-04-28T03:55:14.071598+0000","172.21.15.46:0/3828444952":"2024-04-28T03:54:40.659687+0000","172.21.15.46:0/3592746122":"2024-04-28T03:56:30.716510+0000","172.21.15.46:0/1243800260":"2024-04-28T03:55:14.071598+0000","172.21.15.46:0/3576221786":"2024-04-28T03:55:14.071598+0000","172.21.15.46:0/1611290489":"2024-04-28T03:56:30.716510+0000","172.21.15.46:6800/3824329935":"2024-04-28T03:54:40.659687+0000","172.21.15.46:6800/333567692":"2024-04-28T03:55:14.071598+0000","172.21.15.46:6801/3824329935":"2024-04-28T03:54:40.659687+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T04:00:08.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.499+0000 7f59dc740700 1 -- 172.21.15.46:0/887681511 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f59c006cce0 msgr2=0x7f59c006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:08.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.499+0000 7f59dc740700 1 --2- 172.21.15.46:0/887681511 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f59c006cce0 0x7f59c006f1a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f59d4103940 tx=0x7f59c4008040 comp rx=0 tx=0).stop 2024-04-27T04:00:08.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.499+0000 7f59dc740700 1 -- 172.21.15.46:0/887681511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59d40ff4a0 msgr2=0x7f59d419b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:08.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.499+0000 7f59dc740700 1 --2- 172.21.15.46:0/887681511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59d40ff4a0 0x7f59d419b7f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f59d000bd70 tx=0x7f59d000be50 comp rx=0 tx=0).stop 2024-04-27T04:00:08.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.499+0000 7f59dc740700 1 -- 172.21.15.46:0/887681511 shutdown_connections 2024-04-27T04:00:08.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.499+0000 7f59dc740700 1 --2- 172.21.15.46:0/887681511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59d40ff4a0 0x7f59d419b7f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:08.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.499+0000 7f59dc740700 1 --2- 172.21.15.46:0/887681511 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f59c006cce0 0x7f59c006f1a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:08.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.499+0000 7f59dc740700 1 --2- 172.21.15.46:0/887681511 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f59d40ffe50 0x7f59d419bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:08.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.499+0000 7f59dc740700 1 -- 172.21.15.46:0/887681511 >> 172.21.15.46:0/887681511 conn(0x7f59d4075f30 msgr2=0x7f59d40fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:08.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.499+0000 7f59dc740700 1 -- 172.21.15.46:0/887681511 shutdown_connections 2024-04-27T04:00:08.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:08.499+0000 7f59dc740700 1 -- 172.21.15.46:0/887681511 wait complete. 2024-04-27T04:00:08.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:08 smithi073 ceph-mon[34160]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:09.227 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-27T03:58:45.241461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-27T04:00:09.227 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-27T04:00:09.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:09 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/887681511' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T04:00:09.440 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:09.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:09 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/887681511' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T04:00:10.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:10 smithi046 ceph-mon[26899]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:10.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:10 smithi073 ceph-mon[34160]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.985+0000 7f035af94700 1 -- 172.21.15.46:0/3175273268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354102770 msgr2=0x7f035410ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.985+0000 7f035af94700 1 --2- 172.21.15.46:0/3175273268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354102770 0x7f035410ac60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0348009910 tx=0x7f0348009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.985+0000 7f035af94700 1 -- 172.21.15.46:0/3175273268 shutdown_connections 2024-04-27T04:00:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.985+0000 7f035af94700 1 --2- 172.21.15.46:0/3175273268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354102770 0x7f035410ac60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.985+0000 7f035af94700 1 --2- 172.21.15.46:0/3175273268 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0354101e50 0x7f0354102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.985+0000 7f035af94700 1 -- 172.21.15.46:0/3175273268 >> 172.21.15.46:0/3175273268 conn(0x7f03540fb640 msgr2=0x7f03540fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.985+0000 7f035af94700 1 -- 172.21.15.46:0/3175273268 shutdown_connections 2024-04-27T04:00:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.985+0000 7f035af94700 1 -- 172.21.15.46:0/3175273268 wait complete. 2024-04-27T04:00:10.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.986+0000 7f035af94700 1 Processor -- start 2024-04-27T04:00:10.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.986+0000 7f035af94700 1 -- start start 2024-04-27T04:00:10.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.986+0000 7f035af94700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354101e50 0x7f035419e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:10.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.986+0000 7f035af94700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0354102770 0x7f035419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:10.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.986+0000 7f035af94700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035419f5a0 con 0x7f0354101e50 2024-04-27T04:00:10.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.986+0000 7f035af94700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0354198b20 con 0x7f0354102770 2024-04-27T04:00:10.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.986+0000 7f0358d30700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354101e50 0x7f035419e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:10.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.986+0000 7f0358d30700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354101e50 0x7f035419e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44882/0 (socket says 172.21.15.46:44882) 2024-04-27T04:00:10.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.986+0000 7f0358d30700 1 -- 172.21.15.46:0/2509040157 learned_addr learned my addr 172.21.15.46:0/2509040157 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:10.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.986+0000 7f0353fff700 1 --2- 172.21.15.46:0/2509040157 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0354102770 0x7f035419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:10.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.987+0000 7f0358d30700 1 -- 172.21.15.46:0/2509040157 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0354102770 msgr2=0x7f035419eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:10.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.987+0000 7f0358d30700 1 --2- 172.21.15.46:0/2509040157 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0354102770 0x7f035419eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:10.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.987+0000 7f0358d30700 1 -- 172.21.15.46:0/2509040157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03480095f0 con 0x7f0354101e50 2024-04-27T04:00:10.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.987+0000 7f0353fff700 1 --2- 172.21.15.46:0/2509040157 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0354102770 0x7f035419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T04:00:10.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.987+0000 7f0358d30700 1 --2- 172.21.15.46:0/2509040157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354101e50 0x7f035419e980 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f034400d900 tx=0x7f034400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:10.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.987+0000 7f0351ffb700 1 -- 172.21.15.46:0/2509040157 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03440041f0 con 0x7f0354101e50 2024-04-27T04:00:10.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.987+0000 7f0351ffb700 1 -- 172.21.15.46:0/2509040157 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0344004cf0 con 0x7f0354101e50 2024-04-27T04:00:10.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.987+0000 7f035af94700 1 -- 172.21.15.46:0/2509040157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0354198e00 con 0x7f0354101e50 2024-04-27T04:00:10.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.987+0000 7f0351ffb700 1 -- 172.21.15.46:0/2509040157 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f034400b750 con 0x7f0354101e50 2024-04-27T04:00:10.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.987+0000 7f035af94700 1 -- 172.21.15.46:0/2509040157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0354199350 con 0x7f0354101e50 2024-04-27T04:00:10.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.988+0000 7f035af94700 1 -- 172.21.15.46:0/2509040157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0354069950 con 0x7f0354101e50 2024-04-27T04:00:10.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.989+0000 7f0351ffb700 1 -- 172.21.15.46:0/2509040157 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0344004350 con 0x7f0354101e50 2024-04-27T04:00:10.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.989+0000 7f0351ffb700 1 --2- 172.21.15.46:0/2509040157 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f033c06cce0 0x7f033c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.989+0000 7f0351ffb700 1 -- 172.21.15.46:0/2509040157 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f034408bf50 con 0x7f0354101e50 2024-04-27T04:00:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.989+0000 7f0353fff700 1 --2- 172.21.15.46:0/2509040157 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f033c06cce0 0x7f033c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.990+0000 7f0353fff700 1 --2- 172.21.15.46:0/2509040157 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f033c06cce0 0x7f033c06f1a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0348006010 tx=0x7f034800b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:10.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:10.994+0000 7f0351ffb700 1 -- 172.21.15.46:0/2509040157 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f03440bfc50 con 0x7f0354101e50 2024-04-27T04:00:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:11.194+0000 7f035af94700 1 -- 172.21.15.46:0/2509040157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f035410de10 con 0x7f0354101e50 2024-04-27T04:00:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:11.195+0000 7f0351ffb700 1 -- 172.21.15.46:0/2509040157 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f03440160a0 con 0x7f0354101e50 2024-04-27T04:00:11.197 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-27T04:00:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:11.197+0000 7f035af94700 1 -- 172.21.15.46:0/2509040157 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f033c06cce0 msgr2=0x7f033c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:11.197+0000 7f035af94700 1 --2- 172.21.15.46:0/2509040157 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f033c06cce0 0x7f033c06f1a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0348006010 tx=0x7f034800b540 comp rx=0 tx=0).stop 2024-04-27T04:00:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:11.197+0000 7f035af94700 1 -- 172.21.15.46:0/2509040157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354101e50 msgr2=0x7f035419e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:11.197+0000 7f035af94700 1 --2- 172.21.15.46:0/2509040157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354101e50 0x7f035419e980 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f034400d900 tx=0x7f034400dcc0 comp rx=0 tx=0).stop 2024-04-27T04:00:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:11.197+0000 7f035af94700 1 -- 172.21.15.46:0/2509040157 shutdown_connections 2024-04-27T04:00:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:11.197+0000 7f035af94700 1 --2- 172.21.15.46:0/2509040157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354101e50 0x7f035419e980 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:11.197+0000 7f035af94700 1 --2- 172.21.15.46:0/2509040157 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f033c06cce0 0x7f033c06f1a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:11.197+0000 7f035af94700 1 --2- 172.21.15.46:0/2509040157 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0354102770 0x7f035419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:11.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:11.197+0000 7f035af94700 1 -- 172.21.15.46:0/2509040157 >> 172.21.15.46:0/2509040157 conn(0x7f03540fb640 msgr2=0x7f035410a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:11.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:11.197+0000 7f035af94700 1 -- 172.21.15.46:0/2509040157 shutdown_connections 2024-04-27T04:00:11.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:11.197+0000 7f035af94700 1 -- 172.21.15.46:0/2509040157 wait complete. 2024-04-27T04:00:11.856 INFO:tasks.cephadm:Setting up client nodes... 2024-04-27T04:00:11.856 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-27T04:00:12.068 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:12.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:12 smithi046 ceph-mon[26899]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:12.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:12 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2509040157' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T04:00:12.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:12 smithi073 ceph-mon[34160]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:12.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:12 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/2509040157' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T04:00:13.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.608+0000 7f2025be1700 1 -- 172.21.15.46:0/1119653385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20201016a0 msgr2=0x7f2020103a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:13.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.608+0000 7f2025be1700 1 --2- 172.21.15.46:0/1119653385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20201016a0 0x7f2020103a90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2010009910 tx=0x7f2010009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:13.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.608+0000 7f2025be1700 1 -- 172.21.15.46:0/1119653385 shutdown_connections 2024-04-27T04:00:13.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.608+0000 7f2025be1700 1 --2- 172.21.15.46:0/1119653385 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2020103fd0 0x7f20201063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:13.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.608+0000 7f2025be1700 1 --2- 172.21.15.46:0/1119653385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20201016a0 0x7f2020103a90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:13.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.608+0000 7f2025be1700 1 -- 172.21.15.46:0/1119653385 >> 172.21.15.46:0/1119653385 conn(0x7f20200fb090 msgr2=0x7f20200fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:13.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.608+0000 7f2025be1700 1 -- 172.21.15.46:0/1119653385 shutdown_connections 2024-04-27T04:00:13.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.608+0000 7f2025be1700 1 -- 172.21.15.46:0/1119653385 wait complete. 2024-04-27T04:00:13.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.608+0000 7f2025be1700 1 Processor -- start 2024-04-27T04:00:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.609+0000 7f2025be1700 1 -- start start 2024-04-27T04:00:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.609+0000 7f2025be1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20201016a0 0x7f2020196760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.609+0000 7f2025be1700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2020103fd0 0x7f2020196ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.609+0000 7f2025be1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20201972c0 con 0x7f20201016a0 2024-04-27T04:00:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.609+0000 7f2025be1700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2020197400 con 0x7f2020103fd0 2024-04-27T04:00:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.609+0000 7f201f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20201016a0 0x7f2020196760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.609+0000 7f201f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20201016a0 0x7f2020196760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44910/0 (socket says 172.21.15.46:44910) 2024-04-27T04:00:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.609+0000 7f201f7fe700 1 -- 172.21.15.46:0/2430845517 learned_addr learned my addr 172.21.15.46:0/2430845517 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.609+0000 7f201effd700 1 --2- 172.21.15.46:0/2430845517 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2020103fd0 0x7f2020196ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.609+0000 7f201f7fe700 1 -- 172.21.15.46:0/2430845517 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2020103fd0 msgr2=0x7f2020196ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.609+0000 7f201f7fe700 1 --2- 172.21.15.46:0/2430845517 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2020103fd0 0x7f2020196ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:13.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.609+0000 7f201f7fe700 1 -- 172.21.15.46:0/2430845517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20100095f0 con 0x7f20201016a0 2024-04-27T04:00:13.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.609+0000 7f201f7fe700 1 --2- 172.21.15.46:0/2430845517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20201016a0 0x7f2020196760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2010006010 tx=0x7f201000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:13.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.609+0000 7f201cff9700 1 -- 172.21.15.46:0/2430845517 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f201001d070 con 0x7f20201016a0 2024-04-27T04:00:13.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.610+0000 7f201cff9700 1 -- 172.21.15.46:0/2430845517 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f201000f460 con 0x7f20201016a0 2024-04-27T04:00:13.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.610+0000 7f2025be1700 1 -- 172.21.15.46:0/2430845517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202019bd60 con 0x7f20201016a0 2024-04-27T04:00:13.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.610+0000 7f201cff9700 1 -- 172.21.15.46:0/2430845517 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2010021640 con 0x7f20201016a0 2024-04-27T04:00:13.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.610+0000 7f2025be1700 1 -- 172.21.15.46:0/2430845517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f202019c270 con 0x7f20201016a0 2024-04-27T04:00:13.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.611+0000 7f2025be1700 1 -- 172.21.15.46:0/2430845517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f202004ec20 con 0x7f20201016a0 2024-04-27T04:00:13.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.611+0000 7f201cff9700 1 -- 172.21.15.46:0/2430845517 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f201000fad0 con 0x7f20201016a0 2024-04-27T04:00:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.611+0000 7f201cff9700 1 --2- 172.21.15.46:0/2430845517 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f200806ccf0 0x7f200806f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.612+0000 7f201cff9700 1 -- 172.21.15.46:0/2430845517 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f201008d2a0 con 0x7f20201016a0 2024-04-27T04:00:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.612+0000 7f201effd700 1 --2- 172.21.15.46:0/2430845517 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f200806ccf0 0x7f200806f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.612+0000 7f201effd700 1 --2- 172.21.15.46:0/2430845517 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f200806ccf0 0x7f200806f1b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2014006fd0 tx=0x7f2014008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:13.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.616+0000 7f201cff9700 1 -- 172.21.15.46:0/2430845517 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f202004ec20 con 0x7f20201016a0 2024-04-27T04:00:13.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.904+0000 7f2025be1700 1 -- 172.21.15.46:0/2430845517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f202004ebb0 con 0x7f20201016a0 2024-04-27T04:00:13.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.908+0000 7f201cff9700 1 -- 172.21.15.46:0/2430845517 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f201005b440 con 0x7f20201016a0 2024-04-27T04:00:13.910 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-04-27T04:00:13.910 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBNeCxmBCsNNhAA7eeeicv5fNvP4DAI1Mlv9Q== 2024-04-27T04:00:13.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.910+0000 7f2025be1700 1 -- 172.21.15.46:0/2430845517 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f200806ccf0 msgr2=0x7f200806f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:13.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.910+0000 7f2025be1700 1 --2- 172.21.15.46:0/2430845517 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f200806ccf0 0x7f200806f1b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2014006fd0 tx=0x7f2014008040 comp rx=0 tx=0).stop 2024-04-27T04:00:13.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.910+0000 7f2025be1700 1 -- 172.21.15.46:0/2430845517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20201016a0 msgr2=0x7f2020196760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:13.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.910+0000 7f2025be1700 1 --2- 172.21.15.46:0/2430845517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20201016a0 0x7f2020196760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2010006010 tx=0x7f201000be20 comp rx=0 tx=0).stop 2024-04-27T04:00:13.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.910+0000 7f2025be1700 1 -- 172.21.15.46:0/2430845517 shutdown_connections 2024-04-27T04:00:13.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.910+0000 7f2025be1700 1 --2- 172.21.15.46:0/2430845517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20201016a0 0x7f2020196760 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:13.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.910+0000 7f2025be1700 1 --2- 172.21.15.46:0/2430845517 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f200806ccf0 0x7f200806f1b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:13.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.910+0000 7f2025be1700 1 --2- 172.21.15.46:0/2430845517 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2020103fd0 0x7f2020196ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:13.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.910+0000 7f2025be1700 1 -- 172.21.15.46:0/2430845517 >> 172.21.15.46:0/2430845517 conn(0x7f20200fb090 msgr2=0x7f20200fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:13.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.910+0000 7f2025be1700 1 -- 172.21.15.46:0/2430845517 shutdown_connections 2024-04-27T04:00:13.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:13.911+0000 7f2025be1700 1 -- 172.21.15.46:0/2430845517 wait complete. 2024-04-27T04:00:14.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:14 smithi046 ceph-mon[26899]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:14.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:14 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2430845517' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T04:00:14.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:14 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2430845517' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T04:00:14.521 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T04:00:14.521 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-27T04:00:14.521 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-27T04:00:14.560 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-27T04:00:14.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:14 smithi073 ceph-mon[34160]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:14.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:14 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/2430845517' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T04:00:14.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:14 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/2430845517' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T04:00:14.783 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi073/config 2024-04-27T04:00:16.403 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:16 smithi073 ceph-mon[34160]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:16.404 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:16 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:00:16.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.402+0000 7f67cb9cf700 1 -- 172.21.15.73:0/2585132038 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67c4100d10 msgr2=0x7f67c4101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:16.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.402+0000 7f67cb9cf700 1 --2- 172.21.15.73:0/2585132038 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67c4100d10 0x7f67c4101170 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009910 tx=0x7f67b4009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:16.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.402+0000 7f67cb9cf700 1 -- 172.21.15.73:0/2585132038 shutdown_connections 2024-04-27T04:00:16.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.402+0000 7f67cb9cf700 1 --2- 172.21.15.73:0/2585132038 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67c4100d10 0x7f67c4101170 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:16.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.402+0000 7f67cb9cf700 1 --2- 172.21.15.73:0/2585132038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67c40ffb10 0x7f67c40fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:16.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.402+0000 7f67cb9cf700 1 -- 172.21.15.73:0/2585132038 >> 172.21.15.73:0/2585132038 conn(0x7f67c40fb090 msgr2=0x7f67c40fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:16.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.402+0000 7f67cb9cf700 1 -- 172.21.15.73:0/2585132038 shutdown_connections 2024-04-27T04:00:16.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.402+0000 7f67cb9cf700 1 -- 172.21.15.73:0/2585132038 wait complete. 2024-04-27T04:00:16.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.403+0000 7f67cb9cf700 1 Processor -- start 2024-04-27T04:00:16.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.403+0000 7f67cb9cf700 1 -- start start 2024-04-27T04:00:16.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.403+0000 7f67cb9cf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67c40ffb10 0x7f67c4196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:16.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.403+0000 7f67cb9cf700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67c4100d10 0x7f67c4196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:16.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.403+0000 7f67cb9cf700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c4197270 con 0x7f67c40ffb10 2024-04-27T04:00:16.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.403+0000 7f67cb9cf700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c41973b0 con 0x7f67c4100d10 2024-04-27T04:00:16.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.403+0000 7f67c8f6a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67c4100d10 0x7f67c4196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:16.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.403+0000 7f67c8f6a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67c4100d10 0x7f67c4196c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38316/0 (socket says 172.21.15.73:38316) 2024-04-27T04:00:16.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.403+0000 7f67c8f6a700 1 -- 172.21.15.73:0/1358603717 learned_addr learned my addr 172.21.15.73:0/1358603717 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-04-27T04:00:16.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.403+0000 7f67c976b700 1 --2- 172.21.15.73:0/1358603717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67c40ffb10 0x7f67c4196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:16.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.403+0000 7f67c8f6a700 1 -- 172.21.15.73:0/1358603717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67c40ffb10 msgr2=0x7f67c4196710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:16.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.403+0000 7f67c8f6a700 1 --2- 172.21.15.73:0/1358603717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67c40ffb10 0x7f67c4196710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:16.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.403+0000 7f67c8f6a700 1 -- 172.21.15.73:0/1358603717 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b40095f0 con 0x7f67c4100d10 2024-04-27T04:00:16.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.403+0000 7f67c976b700 1 --2- 172.21.15.73:0/1358603717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67c40ffb10 0x7f67c4196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T04:00:16.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.404+0000 7f67c8f6a700 1 --2- 172.21.15.73:0/1358603717 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67c4100d10 0x7f67c4196c50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f67b40092e0 tx=0x7f67b400be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:16.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.404+0000 7f67ba7fc700 1 -- 172.21.15.73:0/1358603717 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67b401d070 con 0x7f67c4100d10 2024-04-27T04:00:16.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.404+0000 7f67cb9cf700 1 -- 172.21.15.73:0/1358603717 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c419bdb0 con 0x7f67c4100d10 2024-04-27T04:00:16.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.404+0000 7f67ba7fc700 1 -- 172.21.15.73:0/1358603717 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f67b400f460 con 0x7f67c4100d10 2024-04-27T04:00:16.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.404+0000 7f67ba7fc700 1 -- 172.21.15.73:0/1358603717 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67b4021640 con 0x7f67c4100d10 2024-04-27T04:00:16.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.404+0000 7f67cb9cf700 1 -- 172.21.15.73:0/1358603717 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c419c320 con 0x7f67c4100d10 2024-04-27T04:00:16.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.405+0000 7f67cb9cf700 1 -- 172.21.15.73:0/1358603717 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c404ebb0 con 0x7f67c4100d10 2024-04-27T04:00:16.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.405+0000 7f67ba7fc700 1 -- 172.21.15.73:0/1358603717 <== mon.1 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f67b400f5d0 con 0x7f67c4100d10 2024-04-27T04:00:16.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.406+0000 7f67ba7fc700 1 --2- 172.21.15.73:0/1358603717 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f67b006cce0 0x7f67b006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:16.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.406+0000 7f67ba7fc700 1 -- 172.21.15.73:0/1358603717 <== mon.1 v2:172.21.15.73:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f67b408d290 con 0x7f67c4100d10 2024-04-27T04:00:16.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.406+0000 7f67c976b700 1 --2- 172.21.15.73:0/1358603717 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f67b006cce0 0x7f67b006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:16.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.406+0000 7f67c976b700 1 --2- 172.21.15.73:0/1358603717 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f67b006cce0 0x7f67b006f1a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f67c4100b70 tx=0x7f67c0008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:16.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.410+0000 7f67ba7fc700 1 -- 172.21.15.73:0/1358603717 <== mon.1 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f67b405b430 con 0x7f67c4100d10 2024-04-27T04:00:16.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:16 smithi046 ceph-mon[26899]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:16.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:16 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:00:16.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.684+0000 7f67cb9cf700 1 -- 172.21.15.73:0/1358603717 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f67c4108710 con 0x7f67c4100d10 2024-04-27T04:00:16.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.689+0000 7f67ba7fc700 1 -- 172.21.15.73:0/1358603717 <== mon.1 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f67b405afc0 con 0x7f67c4100d10 2024-04-27T04:00:16.691 INFO:teuthology.orchestra.run.smithi073.stdout:[client.1] 2024-04-27T04:00:16.691 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBQeCxm8LfmKBAAPb45SMZgyVDhjdUuEOYlzA== 2024-04-27T04:00:16.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.691+0000 7f67cb9cf700 1 -- 172.21.15.73:0/1358603717 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f67b006cce0 msgr2=0x7f67b006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:16.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.691+0000 7f67cb9cf700 1 --2- 172.21.15.73:0/1358603717 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f67b006cce0 0x7f67b006f1a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f67c4100b70 tx=0x7f67c0008040 comp rx=0 tx=0).stop 2024-04-27T04:00:16.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.691+0000 7f67cb9cf700 1 -- 172.21.15.73:0/1358603717 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67c4100d10 msgr2=0x7f67c4196c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:16.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.691+0000 7f67cb9cf700 1 --2- 172.21.15.73:0/1358603717 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67c4100d10 0x7f67c4196c50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f67b40092e0 tx=0x7f67b400be20 comp rx=0 tx=0).stop 2024-04-27T04:00:16.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.691+0000 7f67cb9cf700 1 -- 172.21.15.73:0/1358603717 shutdown_connections 2024-04-27T04:00:16.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.691+0000 7f67cb9cf700 1 --2- 172.21.15.73:0/1358603717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67c40ffb10 0x7f67c4196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:16.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.691+0000 7f67cb9cf700 1 --2- 172.21.15.73:0/1358603717 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f67b006cce0 0x7f67b006f1a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:16.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.691+0000 7f67cb9cf700 1 --2- 172.21.15.73:0/1358603717 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67c4100d10 0x7f67c4196c50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:16.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.691+0000 7f67cb9cf700 1 -- 172.21.15.73:0/1358603717 >> 172.21.15.73:0/1358603717 conn(0x7f67c40fb090 msgr2=0x7f67c40685c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:16.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.691+0000 7f67cb9cf700 1 -- 172.21.15.73:0/1358603717 shutdown_connections 2024-04-27T04:00:16.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:00:16.692+0000 7f67cb9cf700 1 -- 172.21.15.73:0/1358603717 wait complete. 2024-04-27T04:00:17.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:17 smithi073 ceph-mon[34160]: from='client.? 172.21.15.73:0/1358603717' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T04:00:17.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:17 smithi073 ceph-mon[34160]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T04:00:17.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:17 smithi073 ceph-mon[34160]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T04:00:17.397 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-04-27T04:00:17.397 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-27T04:00:17.397 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-27T04:00:17.474 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-27T04:00:17.474 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-27T04:00:17.474 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-27T04:00:17.500 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:17 smithi046 ceph-mon[26899]: from='client.? 172.21.15.73:0/1358603717' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T04:00:17.500 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:17 smithi046 ceph-mon[26899]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T04:00:17.500 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:17 smithi046 ceph-mon[26899]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T04:00:17.698 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:18.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:18 smithi046 ceph-mon[26899]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:18.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:18 smithi073 ceph-mon[34160]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.422+0000 7fa936d0f700 1 -- 172.21.15.46:0/1229116501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9300ffaa0 msgr2=0x7fa9300fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.422+0000 7fa936d0f700 1 --2- 172.21.15.46:0/1229116501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9300ffaa0 0x7fa9300fff20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa924009910 tx=0x7fa924009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.422+0000 7fa936d0f700 1 -- 172.21.15.46:0/1229116501 shutdown_connections 2024-04-27T04:00:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.422+0000 7fa936d0f700 1 --2- 172.21.15.46:0/1229116501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9300ffaa0 0x7fa9300fff20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.422+0000 7fa936d0f700 1 --2- 172.21.15.46:0/1229116501 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa930105aa0 0x7fa930105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.422+0000 7fa936d0f700 1 -- 172.21.15.46:0/1229116501 >> 172.21.15.46:0/1229116501 conn(0x7fa930076740 msgr2=0x7fa9300fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.422+0000 7fa936d0f700 1 -- 172.21.15.46:0/1229116501 shutdown_connections 2024-04-27T04:00:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.422+0000 7fa936d0f700 1 -- 172.21.15.46:0/1229116501 wait complete. 2024-04-27T04:00:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.423+0000 7fa936d0f700 1 Processor -- start 2024-04-27T04:00:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.423+0000 7fa936d0f700 1 -- start start 2024-04-27T04:00:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.423+0000 7fa936d0f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9300ffaa0 0x7fa93019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.423+0000 7fa936d0f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa930105aa0 0x7fa93019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.423+0000 7fa936d0f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9301968f0 con 0x7fa9300ffaa0 2024-04-27T04:00:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.423+0000 7fa936d0f700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa930196a60 con 0x7fa930105aa0 2024-04-27T04:00:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.423+0000 7fa934aab700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9300ffaa0 0x7fa93019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.423+0000 7fa934aab700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9300ffaa0 0x7fa93019c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44924/0 (socket says 172.21.15.46:44924) 2024-04-27T04:00:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.423+0000 7fa934aab700 1 -- 172.21.15.46:0/2667445792 learned_addr learned my addr 172.21.15.46:0/2667445792 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.423+0000 7fa92ffff700 1 --2- 172.21.15.46:0/2667445792 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa930105aa0 0x7fa93019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.423+0000 7fa934aab700 1 -- 172.21.15.46:0/2667445792 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa930105aa0 msgr2=0x7fa93019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.423+0000 7fa934aab700 1 --2- 172.21.15.46:0/2667445792 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa930105aa0 0x7fa93019cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.423+0000 7fa934aab700 1 -- 172.21.15.46:0/2667445792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9240095f0 con 0x7fa9300ffaa0 2024-04-27T04:00:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.423+0000 7fa934aab700 1 --2- 172.21.15.46:0/2667445792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9300ffaa0 0x7fa93019c840 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa92000d8d0 tx=0x7fa92000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:19.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.424+0000 7fa92dffb700 1 -- 172.21.15.46:0/2667445792 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa920009940 con 0x7fa9300ffaa0 2024-04-27T04:00:19.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.424+0000 7fa936d0f700 1 -- 172.21.15.46:0/2667445792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa930196d40 con 0x7fa9300ffaa0 2024-04-27T04:00:19.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.424+0000 7fa92dffb700 1 -- 172.21.15.46:0/2667445792 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa920010460 con 0x7fa9300ffaa0 2024-04-27T04:00:19.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.424+0000 7fa92dffb700 1 -- 172.21.15.46:0/2667445792 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa92000f5f0 con 0x7fa9300ffaa0 2024-04-27T04:00:19.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.424+0000 7fa936d0f700 1 -- 172.21.15.46:0/2667445792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa930197290 con 0x7fa9300ffaa0 2024-04-27T04:00:19.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.425+0000 7fa936d0f700 1 -- 172.21.15.46:0/2667445792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9301011e0 con 0x7fa9300ffaa0 2024-04-27T04:00:19.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.425+0000 7fa92dffb700 1 -- 172.21.15.46:0/2667445792 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa9200105d0 con 0x7fa9300ffaa0 2024-04-27T04:00:19.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.426+0000 7fa92dffb700 1 --2- 172.21.15.46:0/2667445792 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fa91806ccf0 0x7fa91806f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:19.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.426+0000 7fa92dffb700 1 -- 172.21.15.46:0/2667445792 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa92008bf20 con 0x7fa9300ffaa0 2024-04-27T04:00:19.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.426+0000 7fa92ffff700 1 --2- 172.21.15.46:0/2667445792 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fa91806ccf0 0x7fa91806f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:19.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.426+0000 7fa92ffff700 1 --2- 172.21.15.46:0/2667445792 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fa91806ccf0 0x7fa91806f1b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa930198060 tx=0x7fa92401a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:19.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.431+0000 7fa92dffb700 1 -- 172.21.15.46:0/2667445792 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fa9200bfc50 con 0x7fa9300ffaa0 2024-04-27T04:00:19.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.691+0000 7fa936d0f700 1 -- 172.21.15.46:0/2667445792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fa93010cca0 con 0x7fa9300ffaa0 2024-04-27T04:00:19.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.695+0000 7fa92dffb700 1 -- 172.21.15.46:0/2667445792 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+174011 (secure 0 0 0) 0x7fa9200bd7d0 con 0x7fa9300ffaa0 2024-04-27T04:00:19.698 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:00:19.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.697+0000 7fa936d0f700 1 -- 172.21.15.46:0/2667445792 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fa91806ccf0 msgr2=0x7fa91806f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:19.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.697+0000 7fa936d0f700 1 --2- 172.21.15.46:0/2667445792 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fa91806ccf0 0x7fa91806f1b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa930198060 tx=0x7fa92401a040 comp rx=0 tx=0).stop 2024-04-27T04:00:19.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.697+0000 7fa936d0f700 1 -- 172.21.15.46:0/2667445792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9300ffaa0 msgr2=0x7fa93019c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:19.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.697+0000 7fa936d0f700 1 --2- 172.21.15.46:0/2667445792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9300ffaa0 0x7fa93019c840 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa92000d8d0 tx=0x7fa92000dc90 comp rx=0 tx=0).stop 2024-04-27T04:00:19.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.697+0000 7fa936d0f700 1 -- 172.21.15.46:0/2667445792 shutdown_connections 2024-04-27T04:00:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.697+0000 7fa936d0f700 1 --2- 172.21.15.46:0/2667445792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9300ffaa0 0x7fa93019c840 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.697+0000 7fa936d0f700 1 --2- 172.21.15.46:0/2667445792 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fa91806ccf0 0x7fa91806f1b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.697+0000 7fa936d0f700 1 --2- 172.21.15.46:0/2667445792 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa930105aa0 0x7fa93019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.697+0000 7fa936d0f700 1 -- 172.21.15.46:0/2667445792 >> 172.21.15.46:0/2667445792 conn(0x7fa930076740 msgr2=0x7fa9300fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.697+0000 7fa936d0f700 1 -- 172.21.15.46:0/2667445792 shutdown_connections 2024-04-27T04:00:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:19.697+0000 7fa936d0f700 1 -- 172.21.15.46:0/2667445792 wait complete. 2024-04-27T04:00:20.362 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:20 smithi046 ceph-mon[26899]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:20.362 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:20 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2667445792' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-27T04:00:20.386 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":19,"flags":0,"active_gid":14217,"active_name":"smithi046.qrfdvr","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":4118816988},{"type":"v1","addr":"172.21.15.46:6801","nonce":4118816988}]},"active_addr":"172.21.15.46:6801/4118816988","active_change":"2024-04-27T03:56:30.716618+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14240,"name":"smithi073.awyllt","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.46:8443/","prometheus":"http://172.21.15.46:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":2344419639}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":91070654}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":2850944251}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":2207159230}]}]} 2024-04-27T04:00:20.389 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-27T04:00:20.390 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-27T04:00:20.390 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-27T04:00:20.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:20 smithi073 ceph-mon[34160]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:20.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:20 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/2667445792' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-27T04:00:20.639 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:22.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.369+0000 7f911a38d700 1 -- 172.21.15.46:0/2666854787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9114100e20 msgr2=0x7f9114101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:22.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.369+0000 7f911a38d700 1 --2- 172.21.15.46:0/2666854787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9114100e20 0x7f9114101200 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9104009910 tx=0x7f9104009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:22.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.369+0000 7f911a38d700 1 -- 172.21.15.46:0/2666854787 shutdown_connections 2024-04-27T04:00:22.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.369+0000 7f911a38d700 1 --2- 172.21.15.46:0/2666854787 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91141017d0 0x7f9114105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:22.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.369+0000 7f911a38d700 1 --2- 172.21.15.46:0/2666854787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9114100e20 0x7f9114101200 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:22.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.369+0000 7f911a38d700 1 -- 172.21.15.46:0/2666854787 >> 172.21.15.46:0/2666854787 conn(0x7f91140fc6d0 msgr2=0x7f91140feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:22.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.369+0000 7f911a38d700 1 -- 172.21.15.46:0/2666854787 shutdown_connections 2024-04-27T04:00:22.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.369+0000 7f911a38d700 1 -- 172.21.15.46:0/2666854787 wait complete. 2024-04-27T04:00:22.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.369+0000 7f911a38d700 1 Processor -- start 2024-04-27T04:00:22.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.369+0000 7f911a38d700 1 -- start start 2024-04-27T04:00:22.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.370+0000 7f911a38d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9114100e20 0x7f91140731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:22.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.370+0000 7f911a38d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91141017d0 0x7f9114073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:22.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.370+0000 7f911a38d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9114073c60 con 0x7f91141017d0 2024-04-27T04:00:22.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.370+0000 7f911a38d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9114073da0 con 0x7f9114100e20 2024-04-27T04:00:22.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.370+0000 7f91137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91141017d0 0x7f9114073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:22.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.370+0000 7f9113fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9114100e20 0x7f91140731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:22.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.370+0000 7f91137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91141017d0 0x7f9114073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48028/0 (socket says 172.21.15.46:48028) 2024-04-27T04:00:22.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.370+0000 7f91137fe700 1 -- 172.21.15.46:0/669340049 learned_addr learned my addr 172.21.15.46:0/669340049 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:22.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.370+0000 7f91137fe700 1 -- 172.21.15.46:0/669340049 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9114100e20 msgr2=0x7f91140731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:22.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.370+0000 7f91137fe700 1 --2- 172.21.15.46:0/669340049 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9114100e20 0x7f91140731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:22.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.370+0000 7f91137fe700 1 -- 172.21.15.46:0/669340049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91040095f0 con 0x7f91141017d0 2024-04-27T04:00:22.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.370+0000 7f9113fff700 1 --2- 172.21.15.46:0/669340049 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9114100e20 0x7f91140731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T04:00:22.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.370+0000 7f91137fe700 1 --2- 172.21.15.46:0/669340049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91141017d0 0x7f9114073720 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f910800d900 tx=0x7f910800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:22.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.371+0000 7f91117fa700 1 -- 172.21.15.46:0/669340049 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9108009900 con 0x7f91141017d0 2024-04-27T04:00:22.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.371+0000 7f91117fa700 1 -- 172.21.15.46:0/669340049 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9108010460 con 0x7f91141017d0 2024-04-27T04:00:22.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.371+0000 7f911a38d700 1 -- 172.21.15.46:0/669340049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91141a2d50 con 0x7f91141017d0 2024-04-27T04:00:22.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.371+0000 7f91117fa700 1 -- 172.21.15.46:0/669340049 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f910800f5f0 con 0x7f91141017d0 2024-04-27T04:00:22.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.371+0000 7f911a38d700 1 -- 172.21.15.46:0/669340049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91141a30b0 con 0x7f91141017d0 2024-04-27T04:00:22.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.372+0000 7f911a38d700 1 -- 172.21.15.46:0/669340049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91141091c0 con 0x7f91141017d0 2024-04-27T04:00:22.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.372+0000 7f91117fa700 1 -- 172.21.15.46:0/669340049 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f91080105d0 con 0x7f91141017d0 2024-04-27T04:00:22.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.372+0000 7f91117fa700 1 --2- 172.21.15.46:0/669340049 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f90fc06cce0 0x7f90fc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:22.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.373+0000 7f91117fa700 1 -- 172.21.15.46:0/669340049 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f910808b9b0 con 0x7f91141017d0 2024-04-27T04:00:22.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.373+0000 7f9113fff700 1 --2- 172.21.15.46:0/669340049 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f90fc06cce0 0x7f90fc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:22.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.373+0000 7f9113fff700 1 --2- 172.21.15.46:0/669340049 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f90fc06cce0 0x7f90fc06f1a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9104000c00 tx=0x7f910401a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:22.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.377+0000 7f91117fa700 1 -- 172.21.15.46:0/669340049 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f9108056140 con 0x7f91141017d0 2024-04-27T04:00:22.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:22 smithi046 ceph-mon[26899]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:22.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:22 smithi073 ceph-mon[34160]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:22.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.583+0000 7f911a38d700 1 -- 172.21.15.46:0/669340049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9114074020 con 0x7f91141017d0 2024-04-27T04:00:22.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.584+0000 7f91117fa700 1 -- 172.21.15.46:0/669340049 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11301 (secure 0 0 0) 0x7f9108020090 con 0x7f91141017d0 2024-04-27T04:00:22.585 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:00:22.585 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":40,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","created":"2024-04-27T03:54:08.502636+0000","modified":"2024-04-27T04:00:04.148052+0000","last_up_change":"2024-04-27T04:00:02.972167+0000","last_in_change":"2024-04-27T03:59:43.096341+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T03:58:45.241461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0310053-0895-4608-9021-6c04075a1f9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2222489853},{"type":"v1","addr":"172.21.15.46:6803","nonce":2222489853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2222489853},{"type":"v1","addr":"172.21.15.46:6805","nonce":2222489853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2222489853},{"type":"v1","addr":"172.21.15.46:6809","nonce":2222489853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2222489853},{"type":"v1","addr":"172.21.15.46:6807","nonce":2222489853}]},"public_addr":"172.21.15.46:6803/2222489853","cluster_addr":"172.21.15.46:6805/2222489853","heartbeat_back_addr":"172.21.15.46:6809/2222489853","heartbeat_front_addr":"172.21.15.46:6807/2222489853","state":["exists","up"]},{"osd":1,"uuid":"57d1cc25-d5db-41fd-aa72-e380cedf2225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":1807987418},{"type":"v1","addr":"172.21.15.46:6811","nonce":1807987418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":1807987418},{"type":"v1","addr":"172.21.15.46:6813","nonce":1807987418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":1807987418},{"type":"v1","addr":"172.21.15.46:6817","nonce":1807987418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":1807987418},{"type":"v1","addr":"172.21.15.46:6815","nonce":1807987418}]},"public_addr":"172.21.15.46:6811/1807987418","cluster_addr":"172.21.15.46:6813/1807987418","heartbeat_back_addr":"172.21.15.46:6817/1807987418","heartbeat_front_addr":"172.21.15.46:6815/1807987418","state":["exists","up"]},{"osd":2,"uuid":"addf9aa9-ba34-4443-b239-6df9f7ef68e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":933743760},{"type":"v1","addr":"172.21.15.46:6819","nonce":933743760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":933743760},{"type":"v1","addr":"172.21.15.46:6821","nonce":933743760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":933743760},{"type":"v1","addr":"172.21.15.46:6825","nonce":933743760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":933743760},{"type":"v1","addr":"172.21.15.46:6823","nonce":933743760}]},"public_addr":"172.21.15.46:6819/933743760","cluster_addr":"172.21.15.46:6821/933743760","heartbeat_back_addr":"172.21.15.46:6825/933743760","heartbeat_front_addr":"172.21.15.46:6823/933743760","state":["exists","up"]},{"osd":3,"uuid":"4fa6eb8e-8816-4308-894d-575c57e335d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":2427344041},{"type":"v1","addr":"172.21.15.73:6801","nonce":2427344041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":2427344041},{"type":"v1","addr":"172.21.15.73:6803","nonce":2427344041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":2427344041},{"type":"v1","addr":"172.21.15.73:6807","nonce":2427344041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":2427344041},{"type":"v1","addr":"172.21.15.73:6805","nonce":2427344041}]},"public_addr":"172.21.15.73:6801/2427344041","cluster_addr":"172.21.15.73:6803/2427344041","heartbeat_back_addr":"172.21.15.73:6807/2427344041","heartbeat_front_addr":"172.21.15.73:6805/2427344041","state":["exists","up"]},{"osd":4,"uuid":"fb0a02c1-cc1a-4c2a-a469-0c941b9fe3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":1194795747},{"type":"v1","addr":"172.21.15.73:6809","nonce":1194795747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":1194795747},{"type":"v1","addr":"172.21.15.73:6811","nonce":1194795747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":1194795747},{"type":"v1","addr":"172.21.15.73:6815","nonce":1194795747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":1194795747},{"type":"v1","addr":"172.21.15.73:6813","nonce":1194795747}]},"public_addr":"172.21.15.73:6809/1194795747","cluster_addr":"172.21.15.73:6811/1194795747","heartbeat_back_addr":"172.21.15.73:6815/1194795747","heartbeat_front_addr":"172.21.15.73:6813/1194795747","state":["exists","up"]},{"osd":5,"uuid":"b9d4c4d4-5633-4e9a-ba64-460134d132ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":4277072929},{"type":"v1","addr":"172.21.15.73:6817","nonce":4277072929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":4277072929},{"type":"v1","addr":"172.21.15.73:6819","nonce":4277072929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":4277072929},{"type":"v1","addr":"172.21.15.73:6823","nonce":4277072929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":4277072929},{"type":"v1","addr":"172.21.15.73:6821","nonce":4277072929}]},"public_addr":"172.21.15.73:6817/4277072929","cluster_addr":"172.21.15.73:6819/4277072929","heartbeat_back_addr":"172.21.15.73:6823/4277072929","heartbeat_front_addr":"172.21.15.73:6821/4277072929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:57:47.645830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:58:15.796052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:58:43.469886+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:59:08.833298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:59:34.410852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T04:00:00.927552+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6800/3280787672":"2024-04-28T03:56:30.716510+0000","172.21.15.46:0/1728399059":"2024-04-28T03:56:30.716510+0000","172.21.15.46:6801/3280787672":"2024-04-28T03:56:30.716510+0000","172.21.15.46:0/2377183342":"2024-04-28T03:55:14.071598+0000","172.21.15.46:0/1318513098":"2024-04-28T03:54:40.659687+0000","172.21.15.46:0/2660836044":"2024-04-28T03:54:40.659687+0000","172.21.15.46:6801/333567692":"2024-04-28T03:55:14.071598+0000","172.21.15.46:0/3828444952":"2024-04-28T03:54:40.659687+0000","172.21.15.46:0/3592746122":"2024-04-28T03:56:30.716510+0000","172.21.15.46:0/1243800260":"2024-04-28T03:55:14.071598+0000","172.21.15.46:0/3576221786":"2024-04-28T03:55:14.071598+0000","172.21.15.46:0/1611290489":"2024-04-28T03:56:30.716510+0000","172.21.15.46:6800/3824329935":"2024-04-28T03:54:40.659687+0000","172.21.15.46:6800/333567692":"2024-04-28T03:55:14.071598+0000","172.21.15.46:6801/3824329935":"2024-04-28T03:54:40.659687+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T04:00:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.586+0000 7f911a38d700 1 -- 172.21.15.46:0/669340049 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f90fc06cce0 msgr2=0x7f90fc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.586+0000 7f911a38d700 1 --2- 172.21.15.46:0/669340049 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f90fc06cce0 0x7f90fc06f1a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9104000c00 tx=0x7f910401a040 comp rx=0 tx=0).stop 2024-04-27T04:00:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.586+0000 7f911a38d700 1 -- 172.21.15.46:0/669340049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91141017d0 msgr2=0x7f9114073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.586+0000 7f911a38d700 1 --2- 172.21.15.46:0/669340049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91141017d0 0x7f9114073720 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f910800d900 tx=0x7f910800dc10 comp rx=0 tx=0).stop 2024-04-27T04:00:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.586+0000 7f911a38d700 1 -- 172.21.15.46:0/669340049 shutdown_connections 2024-04-27T04:00:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.586+0000 7f911a38d700 1 --2- 172.21.15.46:0/669340049 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f90fc06cce0 0x7f90fc06f1a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.586+0000 7f911a38d700 1 --2- 172.21.15.46:0/669340049 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9114100e20 0x7f91140731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.586+0000 7f911a38d700 1 --2- 172.21.15.46:0/669340049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91141017d0 0x7f9114073720 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.586+0000 7f911a38d700 1 -- 172.21.15.46:0/669340049 >> 172.21.15.46:0/669340049 conn(0x7f91140fc6d0 msgr2=0x7f91140fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.586+0000 7f911a38d700 1 -- 172.21.15.46:0/669340049 shutdown_connections 2024-04-27T04:00:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:22.586+0000 7f911a38d700 1 -- 172.21.15.46:0/669340049 wait complete. 2024-04-27T04:00:23.285 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-27T04:00:23.286 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-27T04:00:23.497 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:23.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:23 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/669340049' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T04:00:23.619 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:23 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/669340049' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T04:00:24.476 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:24 smithi073 ceph-mon[34160]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:24.643 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:24 smithi046 ceph-mon[26899]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:25.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.150+0000 7f62e0d21700 1 -- 172.21.15.46:0/673724847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0ff350 msgr2=0x7f62dc0ff730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:25.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.150+0000 7f62e0d21700 1 --2- 172.21.15.46:0/673724847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0ff350 0x7f62dc0ff730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f62cc009910 tx=0x7f62cc009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:25.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.150+0000 7f62e0d21700 1 -- 172.21.15.46:0/673724847 shutdown_connections 2024-04-27T04:00:25.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.150+0000 7f62e0d21700 1 --2- 172.21.15.46:0/673724847 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62dc0ffd00 0x7f62dc1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:25.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.150+0000 7f62e0d21700 1 --2- 172.21.15.46:0/673724847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0ff350 0x7f62dc0ff730 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:25.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.150+0000 7f62e0d21700 1 -- 172.21.15.46:0/673724847 >> 172.21.15.46:0/673724847 conn(0x7f62dc075f40 msgr2=0x7f62dc0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:25.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.150+0000 7f62e0d21700 1 -- 172.21.15.46:0/673724847 shutdown_connections 2024-04-27T04:00:25.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.150+0000 7f62e0d21700 1 -- 172.21.15.46:0/673724847 wait complete. 2024-04-27T04:00:25.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.151+0000 7f62e0d21700 1 Processor -- start 2024-04-27T04:00:25.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.151+0000 7f62e0d21700 1 -- start start 2024-04-27T04:00:25.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.151+0000 7f62e0d21700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62dc0ff350 0x7f62dc19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:25.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.151+0000 7f62e0d21700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0ffd00 0x7f62dc19cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:25.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.151+0000 7f62e0d21700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc19d3f0 con 0x7f62dc0ffd00 2024-04-27T04:00:25.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.151+0000 7f62e0d21700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc1969d0 con 0x7f62dc0ff350 2024-04-27T04:00:25.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.151+0000 7f62d9d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0ffd00 0x7f62dc19cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:25.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.151+0000 7f62d9d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0ffd00 0x7f62dc19cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48054/0 (socket says 172.21.15.46:48054) 2024-04-27T04:00:25.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.151+0000 7f62d9d9b700 1 -- 172.21.15.46:0/1118018275 learned_addr learned my addr 172.21.15.46:0/1118018275 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:25.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.151+0000 7f62da59c700 1 --2- 172.21.15.46:0/1118018275 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62dc0ff350 0x7f62dc19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:25.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.151+0000 7f62d9d9b700 1 -- 172.21.15.46:0/1118018275 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62dc0ff350 msgr2=0x7f62dc19c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:25.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.151+0000 7f62d9d9b700 1 --2- 172.21.15.46:0/1118018275 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62dc0ff350 0x7f62dc19c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:25.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.151+0000 7f62d9d9b700 1 -- 172.21.15.46:0/1118018275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc0095f0 con 0x7f62dc0ffd00 2024-04-27T04:00:25.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.152+0000 7f62d9d9b700 1 --2- 172.21.15.46:0/1118018275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0ffd00 0x7f62dc19cdd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c840 tx=0x7f62d000cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:25.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.152+0000 7f62cb7fe700 1 -- 172.21.15.46:0/1118018275 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62d00041f0 con 0x7f62dc0ffd00 2024-04-27T04:00:25.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.152+0000 7f62cb7fe700 1 -- 172.21.15.46:0/1118018275 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62d0004cf0 con 0x7f62dc0ffd00 2024-04-27T04:00:25.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.152+0000 7f62e0d21700 1 -- 172.21.15.46:0/1118018275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62dc196cb0 con 0x7f62dc0ffd00 2024-04-27T04:00:25.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.152+0000 7f62cb7fe700 1 -- 172.21.15.46:0/1118018275 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62d00076d0 con 0x7f62dc0ffd00 2024-04-27T04:00:25.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.152+0000 7f62e0d21700 1 -- 172.21.15.46:0/1118018275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62dc197200 con 0x7f62dc0ffd00 2024-04-27T04:00:25.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.153+0000 7f62e0d21700 1 -- 172.21.15.46:0/1118018275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62dc109230 con 0x7f62dc0ffd00 2024-04-27T04:00:25.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.153+0000 7f62cb7fe700 1 -- 172.21.15.46:0/1118018275 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f62d0004350 con 0x7f62dc0ffd00 2024-04-27T04:00:25.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.154+0000 7f62cb7fe700 1 --2- 172.21.15.46:0/1118018275 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f62c406cce0 0x7f62c406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:25.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.154+0000 7f62da59c700 1 --2- 172.21.15.46:0/1118018275 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f62c406cce0 0x7f62c406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:25.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.154+0000 7f62cb7fe700 1 -- 172.21.15.46:0/1118018275 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f62d008bab0 con 0x7f62dc0ffd00 2024-04-27T04:00:25.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.154+0000 7f62da59c700 1 --2- 172.21.15.46:0/1118018275 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f62c406cce0 0x7f62c406f1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f62cc000c00 tx=0x7f62cc01a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:25.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.159+0000 7f62cb7fe700 1 -- 172.21.15.46:0/1118018275 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f62d0059c50 con 0x7f62dc0ffd00 2024-04-27T04:00:25.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.363+0000 7f62e0d21700 1 -- 172.21.15.46:0/1118018275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62dc197e90 con 0x7f62dc0ffd00 2024-04-27T04:00:25.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.364+0000 7f62cb7fe700 1 -- 172.21.15.46:0/1118018275 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11301 (secure 0 0 0) 0x7f62d00597e0 con 0x7f62dc0ffd00 2024-04-27T04:00:25.366 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:00:25.366 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":40,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","created":"2024-04-27T03:54:08.502636+0000","modified":"2024-04-27T04:00:04.148052+0000","last_up_change":"2024-04-27T04:00:02.972167+0000","last_in_change":"2024-04-27T03:59:43.096341+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T03:58:45.241461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0310053-0895-4608-9021-6c04075a1f9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2222489853},{"type":"v1","addr":"172.21.15.46:6803","nonce":2222489853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2222489853},{"type":"v1","addr":"172.21.15.46:6805","nonce":2222489853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2222489853},{"type":"v1","addr":"172.21.15.46:6809","nonce":2222489853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2222489853},{"type":"v1","addr":"172.21.15.46:6807","nonce":2222489853}]},"public_addr":"172.21.15.46:6803/2222489853","cluster_addr":"172.21.15.46:6805/2222489853","heartbeat_back_addr":"172.21.15.46:6809/2222489853","heartbeat_front_addr":"172.21.15.46:6807/2222489853","state":["exists","up"]},{"osd":1,"uuid":"57d1cc25-d5db-41fd-aa72-e380cedf2225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":1807987418},{"type":"v1","addr":"172.21.15.46:6811","nonce":1807987418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":1807987418},{"type":"v1","addr":"172.21.15.46:6813","nonce":1807987418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":1807987418},{"type":"v1","addr":"172.21.15.46:6817","nonce":1807987418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":1807987418},{"type":"v1","addr":"172.21.15.46:6815","nonce":1807987418}]},"public_addr":"172.21.15.46:6811/1807987418","cluster_addr":"172.21.15.46:6813/1807987418","heartbeat_back_addr":"172.21.15.46:6817/1807987418","heartbeat_front_addr":"172.21.15.46:6815/1807987418","state":["exists","up"]},{"osd":2,"uuid":"addf9aa9-ba34-4443-b239-6df9f7ef68e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":933743760},{"type":"v1","addr":"172.21.15.46:6819","nonce":933743760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":933743760},{"type":"v1","addr":"172.21.15.46:6821","nonce":933743760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":933743760},{"type":"v1","addr":"172.21.15.46:6825","nonce":933743760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":933743760},{"type":"v1","addr":"172.21.15.46:6823","nonce":933743760}]},"public_addr":"172.21.15.46:6819/933743760","cluster_addr":"172.21.15.46:6821/933743760","heartbeat_back_addr":"172.21.15.46:6825/933743760","heartbeat_front_addr":"172.21.15.46:6823/933743760","state":["exists","up"]},{"osd":3,"uuid":"4fa6eb8e-8816-4308-894d-575c57e335d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":2427344041},{"type":"v1","addr":"172.21.15.73:6801","nonce":2427344041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":2427344041},{"type":"v1","addr":"172.21.15.73:6803","nonce":2427344041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":2427344041},{"type":"v1","addr":"172.21.15.73:6807","nonce":2427344041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":2427344041},{"type":"v1","addr":"172.21.15.73:6805","nonce":2427344041}]},"public_addr":"172.21.15.73:6801/2427344041","cluster_addr":"172.21.15.73:6803/2427344041","heartbeat_back_addr":"172.21.15.73:6807/2427344041","heartbeat_front_addr":"172.21.15.73:6805/2427344041","state":["exists","up"]},{"osd":4,"uuid":"fb0a02c1-cc1a-4c2a-a469-0c941b9fe3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":1194795747},{"type":"v1","addr":"172.21.15.73:6809","nonce":1194795747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":1194795747},{"type":"v1","addr":"172.21.15.73:6811","nonce":1194795747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":1194795747},{"type":"v1","addr":"172.21.15.73:6815","nonce":1194795747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":1194795747},{"type":"v1","addr":"172.21.15.73:6813","nonce":1194795747}]},"public_addr":"172.21.15.73:6809/1194795747","cluster_addr":"172.21.15.73:6811/1194795747","heartbeat_back_addr":"172.21.15.73:6815/1194795747","heartbeat_front_addr":"172.21.15.73:6813/1194795747","state":["exists","up"]},{"osd":5,"uuid":"b9d4c4d4-5633-4e9a-ba64-460134d132ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":4277072929},{"type":"v1","addr":"172.21.15.73:6817","nonce":4277072929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":4277072929},{"type":"v1","addr":"172.21.15.73:6819","nonce":4277072929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":4277072929},{"type":"v1","addr":"172.21.15.73:6823","nonce":4277072929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":4277072929},{"type":"v1","addr":"172.21.15.73:6821","nonce":4277072929}]},"public_addr":"172.21.15.73:6817/4277072929","cluster_addr":"172.21.15.73:6819/4277072929","heartbeat_back_addr":"172.21.15.73:6823/4277072929","heartbeat_front_addr":"172.21.15.73:6821/4277072929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:57:47.645830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:58:15.796052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:58:43.469886+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:59:08.833298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:59:34.410852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T04:00:00.927552+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6800/3280787672":"2024-04-28T03:56:30.716510+0000","172.21.15.46:0/1728399059":"2024-04-28T03:56:30.716510+0000","172.21.15.46:6801/3280787672":"2024-04-28T03:56:30.716510+0000","172.21.15.46:0/2377183342":"2024-04-28T03:55:14.071598+0000","172.21.15.46:0/1318513098":"2024-04-28T03:54:40.659687+0000","172.21.15.46:0/2660836044":"2024-04-28T03:54:40.659687+0000","172.21.15.46:6801/333567692":"2024-04-28T03:55:14.071598+0000","172.21.15.46:0/3828444952":"2024-04-28T03:54:40.659687+0000","172.21.15.46:0/3592746122":"2024-04-28T03:56:30.716510+0000","172.21.15.46:0/1243800260":"2024-04-28T03:55:14.071598+0000","172.21.15.46:0/3576221786":"2024-04-28T03:55:14.071598+0000","172.21.15.46:0/1611290489":"2024-04-28T03:56:30.716510+0000","172.21.15.46:6800/3824329935":"2024-04-28T03:54:40.659687+0000","172.21.15.46:6800/333567692":"2024-04-28T03:55:14.071598+0000","172.21.15.46:6801/3824329935":"2024-04-28T03:54:40.659687+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T04:00:25.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.367+0000 7f62e0d21700 1 -- 172.21.15.46:0/1118018275 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f62c406cce0 msgr2=0x7f62c406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:25.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.367+0000 7f62e0d21700 1 --2- 172.21.15.46:0/1118018275 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f62c406cce0 0x7f62c406f1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f62cc000c00 tx=0x7f62cc01a040 comp rx=0 tx=0).stop 2024-04-27T04:00:25.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.367+0000 7f62e0d21700 1 -- 172.21.15.46:0/1118018275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0ffd00 msgr2=0x7f62dc19cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:25.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.367+0000 7f62e0d21700 1 --2- 172.21.15.46:0/1118018275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0ffd00 0x7f62dc19cdd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c840 tx=0x7f62d000cb50 comp rx=0 tx=0).stop 2024-04-27T04:00:25.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.367+0000 7f62e0d21700 1 -- 172.21.15.46:0/1118018275 shutdown_connections 2024-04-27T04:00:25.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.367+0000 7f62e0d21700 1 --2- 172.21.15.46:0/1118018275 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f62c406cce0 0x7f62c406f1a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.367+0000 7f62e0d21700 1 --2- 172.21.15.46:0/1118018275 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62dc0ff350 0x7f62dc19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.367+0000 7f62e0d21700 1 --2- 172.21.15.46:0/1118018275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0ffd00 0x7f62dc19cdd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.367+0000 7f62e0d21700 1 -- 172.21.15.46:0/1118018275 >> 172.21.15.46:0/1118018275 conn(0x7f62dc075f40 msgr2=0x7f62dc104740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.367+0000 7f62e0d21700 1 -- 172.21.15.46:0/1118018275 shutdown_connections 2024-04-27T04:00:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:25.367+0000 7f62e0d21700 1 -- 172.21.15.46:0/1118018275 wait complete. 2024-04-27T04:00:25.926 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-27T04:00:25.927 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-27T04:00:25.927 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-27T04:00:25.927 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-27T04:00:25.928 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-27T04:00:25.928 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-27T04:00:26.190 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:26.453 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:26 smithi046 ceph-mon[26899]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:26.453 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:26 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/1118018275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T04:00:26.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:26 smithi073 ceph-mon[34160]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:26.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:26 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/1118018275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T04:00:26.835 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:26.901 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:26.901 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:26.906 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:26.906 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:28.316 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:28 smithi046 ceph-mon[26899]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:28.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:28 smithi073 ceph-mon[34160]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:30.373 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:30 smithi046 ceph-mon[26899]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:30.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:30 smithi073 ceph-mon[34160]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:31.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.168+0000 7f8ef68ec700 1 -- 172.21.15.46:0/2345776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0101ce0 msgr2=0x7f8ef01020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:31.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.168+0000 7f8ef68ec700 1 --2- 172.21.15.46:0/2345776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0101ce0 0x7f8ef01020c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0009910 tx=0x7f8ee0009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:31.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.168+0000 7f8ef68ec700 1 -- 172.21.15.46:0/2345776 shutdown_connections 2024-04-27T04:00:31.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.168+0000 7f8ef68ec700 1 --2- 172.21.15.46:0/2345776 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ef0102690 0x7f8ef010ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.168+0000 7f8ef68ec700 1 --2- 172.21.15.46:0/2345776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0101ce0 0x7f8ef01020c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.168+0000 7f8ef68ec700 1 -- 172.21.15.46:0/2345776 >> 172.21.15.46:0/2345776 conn(0x7f8ef00fb5e0 msgr2=0x7f8ef00fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:31.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.168+0000 7f8ef68ec700 1 -- 172.21.15.46:0/2345776 shutdown_connections 2024-04-27T04:00:31.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.168+0000 7f8ef68ec700 1 -- 172.21.15.46:0/2345776 wait complete. 2024-04-27T04:00:31.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.168+0000 7f8ef68ec700 1 Processor -- start 2024-04-27T04:00:31.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.169+0000 7f8ef68ec700 1 -- start start 2024-04-27T04:00:31.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.169+0000 7f8ef68ec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0101ce0 0x7f8ef00ff9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:31.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.169+0000 7f8ef68ec700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ef0102690 0x7f8ef00fff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:31.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.169+0000 7f8ef68ec700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef0100450 con 0x7f8ef0101ce0 2024-04-27T04:00:31.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.169+0000 7f8ef68ec700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef0100590 con 0x7f8ef0102690 2024-04-27T04:00:31.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.169+0000 7f8eeffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0101ce0 0x7f8ef00ff9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:31.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.169+0000 7f8eeffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0101ce0 0x7f8ef00ff9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41726/0 (socket says 172.21.15.46:41726) 2024-04-27T04:00:31.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.169+0000 7f8eeffff700 1 -- 172.21.15.46:0/2259691326 learned_addr learned my addr 172.21.15.46:0/2259691326 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:31.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.169+0000 7f8eef7fe700 1 --2- 172.21.15.46:0/2259691326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ef0102690 0x7f8ef00fff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:31.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.169+0000 7f8eeffff700 1 -- 172.21.15.46:0/2259691326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ef0102690 msgr2=0x7f8ef00fff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:31.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.169+0000 7f8eeffff700 1 --2- 172.21.15.46:0/2259691326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ef0102690 0x7f8ef00fff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.169+0000 7f8eeffff700 1 -- 172.21.15.46:0/2259691326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee00095f0 con 0x7f8ef0101ce0 2024-04-27T04:00:31.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.170+0000 7f8eeffff700 1 --2- 172.21.15.46:0/2259691326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0101ce0 0x7f8ef00ff9d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8ee000bd70 tx=0x7f8ee000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:31.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.170+0000 7f8eed7fa700 1 -- 172.21.15.46:0/2259691326 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ee001d070 con 0x7f8ef0101ce0 2024-04-27T04:00:31.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.170+0000 7f8ef68ec700 1 -- 172.21.15.46:0/2259691326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef0100810 con 0x7f8ef0101ce0 2024-04-27T04:00:31.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.170+0000 7f8eed7fa700 1 -- 172.21.15.46:0/2259691326 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ee0005700 con 0x7f8ef0101ce0 2024-04-27T04:00:31.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.170+0000 7f8eed7fa700 1 -- 172.21.15.46:0/2259691326 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ee0017400 con 0x7f8ef0101ce0 2024-04-27T04:00:31.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.170+0000 7f8ef68ec700 1 -- 172.21.15.46:0/2259691326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef01a3070 con 0x7f8ef0101ce0 2024-04-27T04:00:31.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.171+0000 7f8ef68ec700 1 -- 172.21.15.46:0/2259691326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8ef0069730 con 0x7f8ef0101ce0 2024-04-27T04:00:31.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.172+0000 7f8eed7fa700 1 -- 172.21.15.46:0/2259691326 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8ee0003a60 con 0x7f8ef0101ce0 2024-04-27T04:00:31.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.172+0000 7f8eed7fa700 1 --2- 172.21.15.46:0/2259691326 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8ed806cce0 0x7f8ed806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:31.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.172+0000 7f8eed7fa700 1 -- 172.21.15.46:0/2259691326 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8ee008d7c0 con 0x7f8ef0101ce0 2024-04-27T04:00:31.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.172+0000 7f8eef7fe700 1 --2- 172.21.15.46:0/2259691326 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8ed806cce0 0x7f8ed806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:31.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.173+0000 7f8eed7fa700 1 --2- 172.21.15.46:0/2259691326 >> [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760] conn(0x7f8ed80729c0 0x7f8ed8074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:31.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.173+0000 7f8eed7fa700 1 -- 172.21.15.46:0/2259691326 --> [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8ed8075490 con 0x7f8ed80729c0 2024-04-27T04:00:31.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.173+0000 7f8eed7fa700 1 -- 172.21.15.46:0/2259691326 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ee008dc20 con 0x7f8ef0101ce0 2024-04-27T04:00:31.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.173+0000 7f8ef4e89700 1 --2- 172.21.15.46:0/2259691326 >> [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760] conn(0x7f8ed80729c0 0x7f8ed8074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:31.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.173+0000 7f8eef7fe700 1 --2- 172.21.15.46:0/2259691326 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8ed806cce0 0x7f8ed806f1a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0101100 tx=0x7f8ee4009190 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:31.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.173+0000 7f8ef4e89700 1 --2- 172.21.15.46:0/2259691326 >> [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760] conn(0x7f8ed80729c0 0x7f8ed8074de0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:31.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.174+0000 7f8eed7fa700 1 -- 172.21.15.46:0/2259691326 <== osd.2 v2:172.21.15.46:6818/933743760 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f8ed8075490 con 0x7f8ed80729c0 2024-04-27T04:00:31.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.189+0000 7f8ef68ec700 1 -- 172.21.15.46:0/2259691326 --> [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8ef010e090 con 0x7f8ed80729c0 2024-04-27T04:00:31.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.190+0000 7f8eed7fa700 1 -- 172.21.15.46:0/2259691326 <== osd.2 v2:172.21.15.46:6818/933743760 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8ef010e090 con 0x7f8ed80729c0 2024-04-27T04:00:31.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.190+0000 7f8ef68ec700 1 -- 172.21.15.46:0/2259691326 >> [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760] conn(0x7f8ed80729c0 msgr2=0x7f8ed8074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:31.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.190+0000 7f8ef68ec700 1 --2- 172.21.15.46:0/2259691326 >> [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760] conn(0x7f8ed80729c0 0x7f8ed8074de0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.190+0000 7f8ef68ec700 1 -- 172.21.15.46:0/2259691326 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8ed806cce0 msgr2=0x7f8ed806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:31.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.190+0000 7f8ef68ec700 1 --2- 172.21.15.46:0/2259691326 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8ed806cce0 0x7f8ed806f1a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0101100 tx=0x7f8ee4009190 comp rx=0 tx=0).stop 2024-04-27T04:00:31.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.190+0000 7f8ef68ec700 1 -- 172.21.15.46:0/2259691326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0101ce0 msgr2=0x7f8ef00ff9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:31.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.190+0000 7f8ef68ec700 1 --2- 172.21.15.46:0/2259691326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0101ce0 0x7f8ef00ff9d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8ee000bd70 tx=0x7f8ee000bda0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.190+0000 7f8ef68ec700 1 -- 172.21.15.46:0/2259691326 shutdown_connections 2024-04-27T04:00:31.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.190+0000 7f8ef68ec700 1 --2- 172.21.15.46:0/2259691326 >> [v2:172.21.15.46:6818/933743760,v1:172.21.15.46:6819/933743760] conn(0x7f8ed80729c0 0x7f8ed8074de0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.190+0000 7f8ef68ec700 1 --2- 172.21.15.46:0/2259691326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0101ce0 0x7f8ef00ff9d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.190+0000 7f8ef68ec700 1 --2- 172.21.15.46:0/2259691326 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8ed806cce0 0x7f8ed806f1a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.190+0000 7f8ef68ec700 1 --2- 172.21.15.46:0/2259691326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ef0102690 0x7f8ef00fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.190+0000 7f8ef68ec700 1 -- 172.21.15.46:0/2259691326 >> 172.21.15.46:0/2259691326 conn(0x7f8ef00fb5e0 msgr2=0x7f8ef0103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:31.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.190+0000 7f8ef68ec700 1 -- 172.21.15.46:0/2259691326 shutdown_connections 2024-04-27T04:00:31.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.190+0000 7f8ef68ec700 1 -- 172.21.15.46:0/2259691326 wait complete. 2024-04-27T04:00:31.353 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:00:31.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.570+0000 7fae1acb4700 1 -- 172.21.15.46:0/2332052208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae140ffaa0 msgr2=0x7fae140fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:31.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.570+0000 7fae1acb4700 1 --2- 172.21.15.46:0/2332052208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae140ffaa0 0x7fae140fff20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fae00009910 tx=0x7fae00009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:31.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.570+0000 7fae1acb4700 1 -- 172.21.15.46:0/2332052208 shutdown_connections 2024-04-27T04:00:31.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.570+0000 7fae1acb4700 1 --2- 172.21.15.46:0/2332052208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae140ffaa0 0x7fae140fff20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.570+0000 7fae1acb4700 1 --2- 172.21.15.46:0/2332052208 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fae14105aa0 0x7fae14105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.570+0000 7fae1acb4700 1 -- 172.21.15.46:0/2332052208 >> 172.21.15.46:0/2332052208 conn(0x7fae14076740 msgr2=0x7fae140fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:31.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.570+0000 7fae1acb4700 1 -- 172.21.15.46:0/2332052208 shutdown_connections 2024-04-27T04:00:31.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.570+0000 7fae1acb4700 1 -- 172.21.15.46:0/2332052208 wait complete. 2024-04-27T04:00:31.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.571+0000 7fae1acb4700 1 Processor -- start 2024-04-27T04:00:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.571+0000 7fae1acb4700 1 -- start start 2024-04-27T04:00:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.571+0000 7fae1acb4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae140ffaa0 0x7fae1419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.571+0000 7fae1acb4700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fae14105aa0 0x7fae1419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.571+0000 7fae1acb4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1419d320 con 0x7fae140ffaa0 2024-04-27T04:00:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.571+0000 7fae1acb4700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae14196900 con 0x7fae14105aa0 2024-04-27T04:00:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.571+0000 7fae18a50700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae140ffaa0 0x7fae1419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.571+0000 7fae18a50700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae140ffaa0 0x7fae1419c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41750/0 (socket says 172.21.15.46:41750) 2024-04-27T04:00:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.571+0000 7fae18a50700 1 -- 172.21.15.46:0/956543816 learned_addr learned my addr 172.21.15.46:0/956543816 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.571+0000 7fae13fff700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fae14105aa0 0x7fae1419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.572+0000 7fae18a50700 1 -- 172.21.15.46:0/956543816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fae14105aa0 msgr2=0x7fae1419cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.572+0000 7fae18a50700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fae14105aa0 0x7fae1419cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.572+0000 7fae18a50700 1 -- 172.21.15.46:0/956543816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae000095f0 con 0x7fae140ffaa0 2024-04-27T04:00:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.572+0000 7fae13fff700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fae14105aa0 0x7fae1419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T04:00:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.572+0000 7fae18a50700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae140ffaa0 0x7fae1419c7c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fae0800d900 tx=0x7fae0800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.572+0000 7fae11ffb700 1 -- 172.21.15.46:0/956543816 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae080041f0 con 0x7fae140ffaa0 2024-04-27T04:00:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.572+0000 7fae1acb4700 1 -- 172.21.15.46:0/956543816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae14196be0 con 0x7fae140ffaa0 2024-04-27T04:00:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.572+0000 7fae11ffb700 1 -- 172.21.15.46:0/956543816 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae08004cf0 con 0x7fae140ffaa0 2024-04-27T04:00:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.572+0000 7fae11ffb700 1 -- 172.21.15.46:0/956543816 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae0800b750 con 0x7fae140ffaa0 2024-04-27T04:00:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.572+0000 7fae1acb4700 1 -- 172.21.15.46:0/956543816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae14197130 con 0x7fae140ffaa0 2024-04-27T04:00:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.573+0000 7fae1acb4700 1 -- 172.21.15.46:0/956543816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fae1410cfa0 con 0x7fae140ffaa0 2024-04-27T04:00:31.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:31 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:00:31.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.574+0000 7fae11ffb700 1 -- 172.21.15.46:0/956543816 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fae08004350 con 0x7fae140ffaa0 2024-04-27T04:00:31.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.574+0000 7fae11ffb700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fae0406cce0 0x7fae0406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:31.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.574+0000 7fae11ffb700 1 -- 172.21.15.46:0/956543816 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fae0808c010 con 0x7fae140ffaa0 2024-04-27T04:00:31.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.574+0000 7fae13fff700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fae0406cce0 0x7fae0406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:31.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.575+0000 7fae11ffb700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.73:6816/4277072929,v1:172.21.15.73:6817/4277072929] conn(0x7fae040729c0 0x7fae04074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:31.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.575+0000 7fae11ffb700 1 -- 172.21.15.46:0/956543816 --> [v2:172.21.15.73:6816/4277072929,v1:172.21.15.73:6817/4277072929] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae04075490 con 0x7fae040729c0 2024-04-27T04:00:31.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.575+0000 7fae11ffb700 1 -- 172.21.15.46:0/956543816 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae0808e6e0 con 0x7fae140ffaa0 2024-04-27T04:00:31.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.575+0000 7fae19251700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.73:6816/4277072929,v1:172.21.15.73:6817/4277072929] conn(0x7fae040729c0 0x7fae04074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:31.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.575+0000 7fae13fff700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fae0406cce0 0x7fae0406f1a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fae14197ac0 tx=0x7fae0001a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:31.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.575+0000 7fae19251700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.73:6816/4277072929,v1:172.21.15.73:6817/4277072929] conn(0x7fae040729c0 0x7fae04074de0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:31.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.576+0000 7fae11ffb700 1 -- 172.21.15.46:0/956543816 <== osd.5 v2:172.21.15.73:6816/4277072929 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fae04075490 con 0x7fae040729c0 2024-04-27T04:00:31.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.592+0000 7fae1acb4700 1 -- 172.21.15.46:0/956543816 --> [v2:172.21.15.73:6816/4277072929,v1:172.21.15.73:6817/4277072929] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fae14197590 con 0x7fae040729c0 2024-04-27T04:00:31.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.592+0000 7fae11ffb700 1 -- 172.21.15.46:0/956543816 <== osd.5 v2:172.21.15.73:6816/4277072929 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fae14197590 con 0x7fae040729c0 2024-04-27T04:00:31.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.592+0000 7fae1acb4700 1 -- 172.21.15.46:0/956543816 >> [v2:172.21.15.73:6816/4277072929,v1:172.21.15.73:6817/4277072929] conn(0x7fae040729c0 msgr2=0x7fae04074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:31.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.592+0000 7fae1acb4700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.73:6816/4277072929,v1:172.21.15.73:6817/4277072929] conn(0x7fae040729c0 0x7fae04074de0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.592+0000 7fae1acb4700 1 -- 172.21.15.46:0/956543816 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fae0406cce0 msgr2=0x7fae0406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:31.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.593+0000 7fae1acb4700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fae0406cce0 0x7fae0406f1a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fae14197ac0 tx=0x7fae0001a040 comp rx=0 tx=0).stop 2024-04-27T04:00:31.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.593+0000 7fae1acb4700 1 -- 172.21.15.46:0/956543816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae140ffaa0 msgr2=0x7fae1419c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:31.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.593+0000 7fae1acb4700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae140ffaa0 0x7fae1419c7c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fae0800d900 tx=0x7fae0800dcc0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.593+0000 7fae1acb4700 1 -- 172.21.15.46:0/956543816 shutdown_connections 2024-04-27T04:00:31.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.593+0000 7fae1acb4700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.73:6816/4277072929,v1:172.21.15.73:6817/4277072929] conn(0x7fae040729c0 0x7fae04074de0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.593+0000 7fae1acb4700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae140ffaa0 0x7fae1419c7c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.593+0000 7fae1acb4700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fae0406cce0 0x7fae0406f1a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.593+0000 7fae1acb4700 1 --2- 172.21.15.46:0/956543816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fae14105aa0 0x7fae1419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:31.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.593+0000 7fae1acb4700 1 -- 172.21.15.46:0/956543816 >> 172.21.15.46:0/956543816 conn(0x7fae14076740 msgr2=0x7fae14068820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:31.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.593+0000 7fae1acb4700 1 -- 172.21.15.46:0/956543816 shutdown_connections 2024-04-27T04:00:31.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:31.593+0000 7fae1acb4700 1 -- 172.21.15.46:0/956543816 wait complete. 2024-04-27T04:00:32.331 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:32 smithi046 ceph-mon[26899]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:32.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:32 smithi073 ceph-mon[34160]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:34.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:34 smithi046 ceph-mon[26899]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:34.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:34 smithi073 ceph-mon[34160]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.632+0000 7f1f5f482700 1 -- 172.21.15.46:0/3629849772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f58105ab0 msgr2=0x7f1f58105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.632+0000 7f1f5f482700 1 --2- 172.21.15.46:0/3629849772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f58105ab0 0x7f1f58105e90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1f54009910 tx=0x7f1f54009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.632+0000 7f1f5f482700 1 -- 172.21.15.46:0/3629849772 shutdown_connections 2024-04-27T04:00:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.632+0000 7f1f5f482700 1 --2- 172.21.15.46:0/3629849772 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f580ffab0 0x7f1f580fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.632+0000 7f1f5f482700 1 --2- 172.21.15.46:0/3629849772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f58105ab0 0x7f1f58105e90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.632+0000 7f1f5f482700 1 -- 172.21.15.46:0/3629849772 >> 172.21.15.46:0/3629849772 conn(0x7f1f580fb5e0 msgr2=0x7f1f580fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.632+0000 7f1f5f482700 1 -- 172.21.15.46:0/3629849772 shutdown_connections 2024-04-27T04:00:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.632+0000 7f1f5f482700 1 -- 172.21.15.46:0/3629849772 wait complete. 2024-04-27T04:00:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.633+0000 7f1f5f482700 1 Processor -- start 2024-04-27T04:00:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.633+0000 7f1f5f482700 1 -- start start 2024-04-27T04:00:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.633+0000 7f1f5f482700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f580ffab0 0x7f1f58197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.633+0000 7f1f5f482700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f58105ab0 0x7f1f58197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.633+0000 7f1f5f482700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f58198480 con 0x7f1f580ffab0 2024-04-27T04:00:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.633+0000 7f1f5f482700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f581985c0 con 0x7f1f58105ab0 2024-04-27T04:00:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.633+0000 7f1f5d21e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f580ffab0 0x7f1f58197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.633+0000 7f1f5d21e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f580ffab0 0x7f1f58197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41782/0 (socket says 172.21.15.46:41782) 2024-04-27T04:00:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.633+0000 7f1f5d21e700 1 -- 172.21.15.46:0/2105430024 learned_addr learned my addr 172.21.15.46:0/2105430024 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.633+0000 7f1f5ca1d700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f58105ab0 0x7f1f58197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.633+0000 7f1f5d21e700 1 -- 172.21.15.46:0/2105430024 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f58105ab0 msgr2=0x7f1f58197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.633+0000 7f1f5d21e700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f58105ab0 0x7f1f58197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.633+0000 7f1f5d21e700 1 -- 172.21.15.46:0/2105430024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f540095f0 con 0x7f1f580ffab0 2024-04-27T04:00:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.633+0000 7f1f5ca1d700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f58105ab0 0x7f1f58197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T04:00:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.633+0000 7f1f5d21e700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f580ffab0 0x7f1f58197970 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400bd70 tx=0x7f1f5400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.634+0000 7f1f4e7fc700 1 -- 172.21.15.46:0/2105430024 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f5401d070 con 0x7f1f580ffab0 2024-04-27T04:00:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.634+0000 7f1f5f482700 1 -- 172.21.15.46:0/2105430024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f581987e0 con 0x7f1f580ffab0 2024-04-27T04:00:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.634+0000 7f1f4e7fc700 1 -- 172.21.15.46:0/2105430024 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f5400f460 con 0x7f1f580ffab0 2024-04-27T04:00:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.634+0000 7f1f4e7fc700 1 -- 172.21.15.46:0/2105430024 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f54017630 con 0x7f1f580ffab0 2024-04-27T04:00:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.634+0000 7f1f5f482700 1 -- 172.21.15.46:0/2105430024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f581a0d40 con 0x7f1f580ffab0 2024-04-27T04:00:34.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.634+0000 7f1f5f482700 1 -- 172.21.15.46:0/2105430024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1f5810cfb0 con 0x7f1f580ffab0 2024-04-27T04:00:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.635+0000 7f1f4e7fc700 1 -- 172.21.15.46:0/2105430024 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1f5400f5d0 con 0x7f1f580ffab0 2024-04-27T04:00:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.636+0000 7f1f4e7fc700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1f4406cce0 0x7f1f4406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.636+0000 7f1f5ca1d700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1f4406cce0 0x7f1f4406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.636+0000 7f1f4e7fc700 1 -- 172.21.15.46:0/2105430024 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1f5408d6b0 con 0x7f1f580ffab0 2024-04-27T04:00:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.636+0000 7f1f4e7fc700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.73:6808/1194795747,v1:172.21.15.73:6809/1194795747] conn(0x7f1f440729c0 0x7f1f44074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.636+0000 7f1f4e7fc700 1 -- 172.21.15.46:0/2105430024 --> [v2:172.21.15.73:6808/1194795747,v1:172.21.15.73:6809/1194795747] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f44075490 con 0x7f1f440729c0 2024-04-27T04:00:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.636+0000 7f1f4e7fc700 1 -- 172.21.15.46:0/2105430024 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f5408db10 con 0x7f1f580ffab0 2024-04-27T04:00:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.636+0000 7f1f5ca1d700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1f4406cce0 0x7f1f4406f1a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1f58069660 tx=0x7f1f48005f50 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.636+0000 7f1f5da1f700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.73:6808/1194795747,v1:172.21.15.73:6809/1194795747] conn(0x7f1f440729c0 0x7f1f44074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.637+0000 7f1f5da1f700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.73:6808/1194795747,v1:172.21.15.73:6809/1194795747] conn(0x7f1f440729c0 0x7f1f44074de0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.638+0000 7f1f4e7fc700 1 -- 172.21.15.46:0/2105430024 <== osd.4 v2:172.21.15.73:6808/1194795747 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1f44075490 con 0x7f1f440729c0 2024-04-27T04:00:34.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.653+0000 7f1f5f482700 1 -- 172.21.15.46:0/2105430024 --> [v2:172.21.15.73:6808/1194795747,v1:172.21.15.73:6809/1194795747] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1f581987e0 con 0x7f1f440729c0 2024-04-27T04:00:34.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.654+0000 7f1f4e7fc700 1 -- 172.21.15.46:0/2105430024 <== osd.4 v2:172.21.15.73:6808/1194795747 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1f581987e0 con 0x7f1f440729c0 2024-04-27T04:00:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.654+0000 7f1f5f482700 1 -- 172.21.15.46:0/2105430024 >> [v2:172.21.15.73:6808/1194795747,v1:172.21.15.73:6809/1194795747] conn(0x7f1f440729c0 msgr2=0x7f1f44074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.654+0000 7f1f5f482700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.73:6808/1194795747,v1:172.21.15.73:6809/1194795747] conn(0x7f1f440729c0 0x7f1f44074de0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.654+0000 7f1f5f482700 1 -- 172.21.15.46:0/2105430024 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1f4406cce0 msgr2=0x7f1f4406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.654+0000 7f1f5f482700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1f4406cce0 0x7f1f4406f1a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1f58069660 tx=0x7f1f48005f50 comp rx=0 tx=0).stop 2024-04-27T04:00:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.654+0000 7f1f5f482700 1 -- 172.21.15.46:0/2105430024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f580ffab0 msgr2=0x7f1f58197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.654+0000 7f1f5f482700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f580ffab0 0x7f1f58197970 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400bd70 tx=0x7f1f5400be50 comp rx=0 tx=0).stop 2024-04-27T04:00:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.654+0000 7f1f5f482700 1 -- 172.21.15.46:0/2105430024 shutdown_connections 2024-04-27T04:00:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.654+0000 7f1f5f482700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.73:6808/1194795747,v1:172.21.15.73:6809/1194795747] conn(0x7f1f440729c0 0x7f1f44074de0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.654+0000 7f1f5f482700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f580ffab0 0x7f1f58197970 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.654+0000 7f1f5f482700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1f4406cce0 0x7f1f4406f1a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.654+0000 7f1f5f482700 1 --2- 172.21.15.46:0/2105430024 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f58105ab0 0x7f1f58197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.654+0000 7f1f5f482700 1 -- 172.21.15.46:0/2105430024 >> 172.21.15.46:0/2105430024 conn(0x7f1f580fb5e0 msgr2=0x7f1f580fc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.654+0000 7f1f5f482700 1 -- 172.21.15.46:0/2105430024 shutdown_connections 2024-04-27T04:00:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.654+0000 7f1f5f482700 1 -- 172.21.15.46:0/2105430024 wait complete. 2024-04-27T04:00:34.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.538+0000 7f29458be700 1 -- 172.21.15.46:0/2979728843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400ffaa0 msgr2=0x7f29400fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:34.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.538+0000 7f29458be700 1 --2- 172.21.15.46:0/2979728843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400ffaa0 0x7f29400fff20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2934009910 tx=0x7f2934009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:34.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.538+0000 7f29458be700 1 -- 172.21.15.46:0/2979728843 shutdown_connections 2024-04-27T04:00:34.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.538+0000 7f29458be700 1 --2- 172.21.15.46:0/2979728843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400ffaa0 0x7f29400fff20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:34.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.538+0000 7f29458be700 1 --2- 172.21.15.46:0/2979728843 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2940105aa0 0x7f2940105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:34.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.538+0000 7f29458be700 1 -- 172.21.15.46:0/2979728843 >> 172.21.15.46:0/2979728843 conn(0x7f2940076740 msgr2=0x7f29400fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:34.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.538+0000 7f29458be700 1 -- 172.21.15.46:0/2979728843 shutdown_connections 2024-04-27T04:00:34.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.538+0000 7f29458be700 1 -- 172.21.15.46:0/2979728843 wait complete. 2024-04-27T04:00:34.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.539+0000 7f29458be700 1 Processor -- start 2024-04-27T04:00:34.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.539+0000 7f29458be700 1 -- start start 2024-04-27T04:00:34.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.539+0000 7f29458be700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f29400ffaa0 0x7f2940197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:34.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.539+0000 7f29458be700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2940105aa0 0x7f2940197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:34.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.539+0000 7f29458be700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2940198410 con 0x7f2940105aa0 2024-04-27T04:00:34.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.539+0000 7f29458be700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2940198550 con 0x7f29400ffaa0 2024-04-27T04:00:34.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.539+0000 7f293e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2940105aa0 0x7f2940197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:34.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.539+0000 7f293e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2940105aa0 0x7f2940197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41766/0 (socket says 172.21.15.46:41766) 2024-04-27T04:00:34.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.539+0000 7f293e7fc700 1 -- 172.21.15.46:0/3847002973 learned_addr learned my addr 172.21.15.46:0/3847002973 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:34.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.539+0000 7f293effd700 1 --2- 172.21.15.46:0/3847002973 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f29400ffaa0 0x7f2940197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:34.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.539+0000 7f293e7fc700 1 -- 172.21.15.46:0/3847002973 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f29400ffaa0 msgr2=0x7f2940197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:34.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.539+0000 7f293e7fc700 1 --2- 172.21.15.46:0/3847002973 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f29400ffaa0 0x7f2940197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:34.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.539+0000 7f293e7fc700 1 -- 172.21.15.46:0/3847002973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29340095f0 con 0x7f2940105aa0 2024-04-27T04:00:34.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.540+0000 7f293e7fc700 1 --2- 172.21.15.46:0/3847002973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2940105aa0 0x7f2940197e40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2934000c00 tx=0x7f293400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:34.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.540+0000 7f29448bc700 1 -- 172.21.15.46:0/3847002973 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f293401d070 con 0x7f2940105aa0 2024-04-27T04:00:34.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.540+0000 7f29448bc700 1 -- 172.21.15.46:0/3847002973 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29340056e0 con 0x7f2940105aa0 2024-04-27T04:00:34.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.540+0000 7f29458be700 1 -- 172.21.15.46:0/3847002973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2940198770 con 0x7f2940105aa0 2024-04-27T04:00:34.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.540+0000 7f29448bc700 1 -- 172.21.15.46:0/3847002973 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f293400fe40 con 0x7f2940105aa0 2024-04-27T04:00:34.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.540+0000 7f29458be700 1 -- 172.21.15.46:0/3847002973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29401a0cd0 con 0x7f2940105aa0 2024-04-27T04:00:34.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.541+0000 7f29458be700 1 -- 172.21.15.46:0/3847002973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f294010cfa0 con 0x7f2940105aa0 2024-04-27T04:00:34.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.541+0000 7f29448bc700 1 -- 172.21.15.46:0/3847002973 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f293400f680 con 0x7f2940105aa0 2024-04-27T04:00:34.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.542+0000 7f29448bc700 1 --2- 172.21.15.46:0/3847002973 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f292806cc90 0x7f292806f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:34.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.542+0000 7f29448bc700 1 -- 172.21.15.46:0/3847002973 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f293408d0f0 con 0x7f2940105aa0 2024-04-27T04:00:34.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.542+0000 7f293effd700 1 --2- 172.21.15.46:0/3847002973 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f292806cc90 0x7f292806f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:34.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.542+0000 7f29448bc700 1 --2- 172.21.15.46:0/3847002973 >> [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418] conn(0x7f2928072970 0x7f2928074d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:34.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.542+0000 7f29448bc700 1 -- 172.21.15.46:0/3847002973 --> [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2928075440 con 0x7f2928072970 2024-04-27T04:00:34.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.542+0000 7f29448bc700 1 -- 172.21.15.46:0/3847002973 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f293408f7c0 con 0x7f2940105aa0 2024-04-27T04:00:34.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.542+0000 7f293f7fe700 1 --2- 172.21.15.46:0/3847002973 >> [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418] conn(0x7f2928072970 0x7f2928074d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:34.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.542+0000 7f293effd700 1 --2- 172.21.15.46:0/3847002973 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f292806cc90 0x7f292806f150 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2930005950 tx=0x7f293000b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:34.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.543+0000 7f293f7fe700 1 --2- 172.21.15.46:0/3847002973 >> [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418] conn(0x7f2928072970 0x7f2928074d90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:34.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.543+0000 7f29448bc700 1 -- 172.21.15.46:0/3847002973 <== osd.1 v2:172.21.15.46:6810/1807987418 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2928075440 con 0x7f2928072970 2024-04-27T04:00:34.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.559+0000 7f29458be700 1 -- 172.21.15.46:0/3847002973 --> [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2940061560 con 0x7f2928072970 2024-04-27T04:00:34.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.559+0000 7f29448bc700 1 -- 172.21.15.46:0/3847002973 <== osd.1 v2:172.21.15.46:6810/1807987418 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2940061560 con 0x7f2928072970 2024-04-27T04:00:34.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.559+0000 7f29458be700 1 -- 172.21.15.46:0/3847002973 >> [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418] conn(0x7f2928072970 msgr2=0x7f2928074d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:34.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.559+0000 7f29458be700 1 --2- 172.21.15.46:0/3847002973 >> [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418] conn(0x7f2928072970 0x7f2928074d90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:34.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.560+0000 7f29458be700 1 -- 172.21.15.46:0/3847002973 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f292806cc90 msgr2=0x7f292806f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:34.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.560+0000 7f29458be700 1 --2- 172.21.15.46:0/3847002973 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f292806cc90 0x7f292806f150 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2930005950 tx=0x7f293000b410 comp rx=0 tx=0).stop 2024-04-27T04:00:34.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.560+0000 7f29458be700 1 -- 172.21.15.46:0/3847002973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2940105aa0 msgr2=0x7f2940197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:34.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.560+0000 7f29458be700 1 --2- 172.21.15.46:0/3847002973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2940105aa0 0x7f2940197e40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2934000c00 tx=0x7f293400bd70 comp rx=0 tx=0).stop 2024-04-27T04:00:34.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.560+0000 7f29458be700 1 -- 172.21.15.46:0/3847002973 shutdown_connections 2024-04-27T04:00:34.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.560+0000 7f29458be700 1 --2- 172.21.15.46:0/3847002973 >> [v2:172.21.15.46:6810/1807987418,v1:172.21.15.46:6811/1807987418] conn(0x7f2928072970 0x7f2928074d90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:34.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.560+0000 7f29458be700 1 --2- 172.21.15.46:0/3847002973 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f292806cc90 0x7f292806f150 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:34.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.560+0000 7f29458be700 1 --2- 172.21.15.46:0/3847002973 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f29400ffaa0 0x7f2940197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:34.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.560+0000 7f29458be700 1 --2- 172.21.15.46:0/3847002973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2940105aa0 0x7f2940197e40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:34.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.560+0000 7f29458be700 1 -- 172.21.15.46:0/3847002973 >> 172.21.15.46:0/3847002973 conn(0x7f2940076740 msgr2=0x7f29400fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:34.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.560+0000 7f29458be700 1 -- 172.21.15.46:0/3847002973 shutdown_connections 2024-04-27T04:00:34.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:34.560+0000 7f29458be700 1 -- 172.21.15.46:0/3847002973 wait complete. 2024-04-27T04:00:35.500 INFO:teuthology.orchestra.run.smithi046.stdout:146028888078 2024-04-27T04:00:35.501 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-27T04:00:35.947 INFO:teuthology.orchestra.run.smithi046.stdout:81604378649 2024-04-27T04:00:35.947 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-27T04:00:36.385 INFO:teuthology.orchestra.run.smithi046.stdout:60129542175 2024-04-27T04:00:36.385 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-27T04:00:36.470 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:36 smithi046 ceph-mon[26899]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:36.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:36 smithi073 ceph-mon[34160]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:36.990 INFO:teuthology.orchestra.run.smithi046.stdout:167503724553 2024-04-27T04:00:36.990 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-27T04:00:37.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.206+0000 7fe06ee96700 1 -- 172.21.15.46:0/2864356466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe068102690 msgr2=0x7fe06810ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:37.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.206+0000 7fe06ee96700 1 --2- 172.21.15.46:0/2864356466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe068102690 0x7fe06810ab80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe054009910 tx=0x7fe054009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:37.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.207+0000 7fe06ee96700 1 -- 172.21.15.46:0/2864356466 shutdown_connections 2024-04-27T04:00:37.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.207+0000 7fe06ee96700 1 --2- 172.21.15.46:0/2864356466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe068102690 0x7fe06810ab80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:37.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.207+0000 7fe06ee96700 1 --2- 172.21.15.46:0/2864356466 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe068101ce0 0x7fe0681020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:37.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.207+0000 7fe06ee96700 1 -- 172.21.15.46:0/2864356466 >> 172.21.15.46:0/2864356466 conn(0x7fe0680fb5e0 msgr2=0x7fe0680fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:37.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.207+0000 7fe06ee96700 1 -- 172.21.15.46:0/2864356466 shutdown_connections 2024-04-27T04:00:37.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.207+0000 7fe06ee96700 1 -- 172.21.15.46:0/2864356466 wait complete. 2024-04-27T04:00:37.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.207+0000 7fe06ee96700 1 Processor -- start 2024-04-27T04:00:37.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.207+0000 7fe06ee96700 1 -- start start 2024-04-27T04:00:37.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.208+0000 7fe06ee96700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe068101ce0 0x7fe0680ffa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:37.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.208+0000 7fe06ee96700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe068102690 0x7fe0680fff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:37.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.208+0000 7fe06ee96700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe068101690 con 0x7fe068102690 2024-04-27T04:00:37.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.208+0000 7fe06ee96700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe068101800 con 0x7fe068101ce0 2024-04-27T04:00:37.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.208+0000 7fe067fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe068102690 0x7fe0680fff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:37.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.208+0000 7fe067fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe068102690 0x7fe0680fff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41806/0 (socket says 172.21.15.46:41806) 2024-04-27T04:00:37.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.208+0000 7fe067fff700 1 -- 172.21.15.46:0/3998928295 learned_addr learned my addr 172.21.15.46:0/3998928295 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:37.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.208+0000 7fe06cc32700 1 --2- 172.21.15.46:0/3998928295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe068101ce0 0x7fe0680ffa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:37.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.208+0000 7fe067fff700 1 -- 172.21.15.46:0/3998928295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe068101ce0 msgr2=0x7fe0680ffa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:37.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.208+0000 7fe067fff700 1 --2- 172.21.15.46:0/3998928295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe068101ce0 0x7fe0680ffa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:37.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.208+0000 7fe067fff700 1 -- 172.21.15.46:0/3998928295 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0540095f0 con 0x7fe068102690 2024-04-27T04:00:37.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.208+0000 7fe067fff700 1 --2- 172.21.15.46:0/3998928295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe068102690 0x7fe0680fff70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe0540092e0 tx=0x7fe05400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:37.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.209+0000 7fe065ffb700 1 -- 172.21.15.46:0/3998928295 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe05401d070 con 0x7fe068102690 2024-04-27T04:00:37.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.209+0000 7fe065ffb700 1 -- 172.21.15.46:0/3998928295 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe05400f460 con 0x7fe068102690 2024-04-27T04:00:37.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.209+0000 7fe06ee96700 1 -- 172.21.15.46:0/3998928295 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0681004b0 con 0x7fe068102690 2024-04-27T04:00:37.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.209+0000 7fe065ffb700 1 -- 172.21.15.46:0/3998928295 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe054017630 con 0x7fe068102690 2024-04-27T04:00:37.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.209+0000 7fe06ee96700 1 -- 172.21.15.46:0/3998928295 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0681a7420 con 0x7fe068102690 2024-04-27T04:00:37.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.210+0000 7fe06ee96700 1 -- 172.21.15.46:0/3998928295 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe06810dff0 con 0x7fe068102690 2024-04-27T04:00:37.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.211+0000 7fe065ffb700 1 -- 172.21.15.46:0/3998928295 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe054017860 con 0x7fe068102690 2024-04-27T04:00:37.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.211+0000 7fe065ffb700 1 --2- 172.21.15.46:0/3998928295 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe05806cce0 0x7fe05806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:37.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.212+0000 7fe065ffb700 1 -- 172.21.15.46:0/3998928295 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe05408d200 con 0x7fe068102690 2024-04-27T04:00:37.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.212+0000 7fe06cc32700 1 --2- 172.21.15.46:0/3998928295 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe05806cce0 0x7fe05806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:37.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.212+0000 7fe065ffb700 1 --2- 172.21.15.46:0/3998928295 >> [v2:172.21.15.73:6800/2427344041,v1:172.21.15.73:6801/2427344041] conn(0x7fe0580729c0 0x7fe058074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:37.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.212+0000 7fe065ffb700 1 -- 172.21.15.46:0/3998928295 --> [v2:172.21.15.73:6800/2427344041,v1:172.21.15.73:6801/2427344041] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe058075490 con 0x7fe0580729c0 2024-04-27T04:00:37.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.212+0000 7fe065ffb700 1 -- 172.21.15.46:0/3998928295 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe05408f8d0 con 0x7fe068102690 2024-04-27T04:00:37.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.212+0000 7fe06d433700 1 --2- 172.21.15.46:0/3998928295 >> [v2:172.21.15.73:6800/2427344041,v1:172.21.15.73:6801/2427344041] conn(0x7fe0580729c0 0x7fe058074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:37.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.212+0000 7fe06cc32700 1 --2- 172.21.15.46:0/3998928295 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe05806cce0 0x7fe05806f1a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00ac50 tx=0x7fe05c00a380 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:37.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.213+0000 7fe06d433700 1 --2- 172.21.15.46:0/3998928295 >> [v2:172.21.15.73:6800/2427344041,v1:172.21.15.73:6801/2427344041] conn(0x7fe0580729c0 0x7fe058074de0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:37.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.214+0000 7fe065ffb700 1 -- 172.21.15.46:0/3998928295 <== osd.3 v2:172.21.15.73:6800/2427344041 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe058075490 con 0x7fe0580729c0 2024-04-27T04:00:37.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.233+0000 7fe06ee96700 1 -- 172.21.15.46:0/3998928295 --> [v2:172.21.15.73:6800/2427344041,v1:172.21.15.73:6801/2427344041] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe068061560 con 0x7fe0580729c0 2024-04-27T04:00:37.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.234+0000 7fe065ffb700 1 -- 172.21.15.46:0/3998928295 <== osd.3 v2:172.21.15.73:6800/2427344041 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe068061560 con 0x7fe0580729c0 2024-04-27T04:00:37.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.234+0000 7fe06ee96700 1 -- 172.21.15.46:0/3998928295 >> [v2:172.21.15.73:6800/2427344041,v1:172.21.15.73:6801/2427344041] conn(0x7fe0580729c0 msgr2=0x7fe058074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:37.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.234+0000 7fe06ee96700 1 --2- 172.21.15.46:0/3998928295 >> [v2:172.21.15.73:6800/2427344041,v1:172.21.15.73:6801/2427344041] conn(0x7fe0580729c0 0x7fe058074de0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.234+0000 7fe06ee96700 1 -- 172.21.15.46:0/3998928295 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe05806cce0 msgr2=0x7fe05806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.234+0000 7fe06ee96700 1 --2- 172.21.15.46:0/3998928295 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe05806cce0 0x7fe05806f1a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00ac50 tx=0x7fe05c00a380 comp rx=0 tx=0).stop 2024-04-27T04:00:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.234+0000 7fe06ee96700 1 -- 172.21.15.46:0/3998928295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe068102690 msgr2=0x7fe0680fff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.234+0000 7fe06ee96700 1 --2- 172.21.15.46:0/3998928295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe068102690 0x7fe0680fff70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe0540092e0 tx=0x7fe05400bd70 comp rx=0 tx=0).stop 2024-04-27T04:00:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.235+0000 7fe06ee96700 1 -- 172.21.15.46:0/3998928295 shutdown_connections 2024-04-27T04:00:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.235+0000 7fe06ee96700 1 --2- 172.21.15.46:0/3998928295 >> [v2:172.21.15.73:6800/2427344041,v1:172.21.15.73:6801/2427344041] conn(0x7fe0580729c0 0x7fe058074de0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.235+0000 7fe06ee96700 1 --2- 172.21.15.46:0/3998928295 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe05806cce0 0x7fe05806f1a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.235+0000 7fe06ee96700 1 --2- 172.21.15.46:0/3998928295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe068101ce0 0x7fe0680ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.235+0000 7fe06ee96700 1 --2- 172.21.15.46:0/3998928295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe068102690 0x7fe0680fff70 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.235+0000 7fe06ee96700 1 -- 172.21.15.46:0/3998928295 >> 172.21.15.46:0/3998928295 conn(0x7fe0680fb5e0 msgr2=0x7fe068103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.235+0000 7fe06ee96700 1 -- 172.21.15.46:0/3998928295 shutdown_connections 2024-04-27T04:00:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:37.235+0000 7fe06ee96700 1 -- 172.21.15.46:0/3998928295 wait complete. 2024-04-27T04:00:37.446 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:37.961 INFO:teuthology.orchestra.run.smithi046.stdout:111669149716 2024-04-27T04:00:37.961 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-27T04:00:37.998 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:38.459 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:38 smithi046 ceph-mon[26899]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:38.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:38 smithi073 ceph-mon[34160]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:40.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.223+0000 7f3cd99dc700 1 -- 172.21.15.46:0/3150337087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd40ffad0 msgr2=0x7f3cd40fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:40.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.223+0000 7f3cd99dc700 1 --2- 172.21.15.46:0/3150337087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd40ffad0 0x7f3cd40fff50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8009910 tx=0x7f3cc8009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:40.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.223+0000 7f3cd99dc700 1 -- 172.21.15.46:0/3150337087 shutdown_connections 2024-04-27T04:00:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.223+0000 7f3cd99dc700 1 --2- 172.21.15.46:0/3150337087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd40ffad0 0x7f3cd40fff50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.223+0000 7f3cd99dc700 1 --2- 172.21.15.46:0/3150337087 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3cd4105b60 0x7f3cd4105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.223+0000 7f3cd99dc700 1 -- 172.21.15.46:0/3150337087 >> 172.21.15.46:0/3150337087 conn(0x7f3cd40fb5e0 msgr2=0x7f3cd40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.223+0000 7f3cd99dc700 1 -- 172.21.15.46:0/3150337087 shutdown_connections 2024-04-27T04:00:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.223+0000 7f3cd99dc700 1 -- 172.21.15.46:0/3150337087 wait complete. 2024-04-27T04:00:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.224+0000 7f3cd99dc700 1 Processor -- start 2024-04-27T04:00:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.224+0000 7f3cd99dc700 1 -- start start 2024-04-27T04:00:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.224+0000 7f3cd99dc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3cd40ffad0 0x7f3cd4197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.224+0000 7f3cd99dc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd4105b60 0x7f3cd4197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.224+0000 7f3cd99dc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd4198460 con 0x7f3cd4105b60 2024-04-27T04:00:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.224+0000 7f3cd99dc700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd41985a0 con 0x7f3cd40ffad0 2024-04-27T04:00:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.224+0000 7f3cd27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd4105b60 0x7f3cd4197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:40.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.224+0000 7f3cd27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd4105b60 0x7f3cd4197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41828/0 (socket says 172.21.15.46:41828) 2024-04-27T04:00:40.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.224+0000 7f3cd27fc700 1 -- 172.21.15.46:0/3341296161 learned_addr learned my addr 172.21.15.46:0/3341296161 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:40.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.224+0000 7f3cd2ffd700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3cd40ffad0 0x7f3cd4197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:40.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.225+0000 7f3cd27fc700 1 -- 172.21.15.46:0/3341296161 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3cd40ffad0 msgr2=0x7f3cd4197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:40.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.225+0000 7f3cd27fc700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3cd40ffad0 0x7f3cd4197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:40.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.225+0000 7f3cd27fc700 1 -- 172.21.15.46:0/3341296161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc80095f0 con 0x7f3cd4105b60 2024-04-27T04:00:40.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.225+0000 7f3cd2ffd700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3cd40ffad0 0x7f3cd4197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T04:00:40.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.225+0000 7f3cd27fc700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd4105b60 0x7f3cd4197e90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3cc800bd70 tx=0x7f3cc800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:40.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.225+0000 7f3cd89da700 1 -- 172.21.15.46:0/3341296161 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cc801d070 con 0x7f3cd4105b60 2024-04-27T04:00:40.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.225+0000 7f3cd99dc700 1 -- 172.21.15.46:0/3341296161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd41a09c0 con 0x7f3cd4105b60 2024-04-27T04:00:40.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.225+0000 7f3cd89da700 1 -- 172.21.15.46:0/3341296161 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3cc80056e0 con 0x7f3cd4105b60 2024-04-27T04:00:40.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.225+0000 7f3cd89da700 1 -- 172.21.15.46:0/3341296161 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cc8017400 con 0x7f3cd4105b60 2024-04-27T04:00:40.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.225+0000 7f3cd99dc700 1 -- 172.21.15.46:0/3341296161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd41a0d20 con 0x7f3cd4105b60 2024-04-27T04:00:40.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.226+0000 7f3cd99dc700 1 -- 172.21.15.46:0/3341296161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3cd410cfd0 con 0x7f3cd4105b60 2024-04-27T04:00:40.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.226+0000 7f3cd89da700 1 -- 172.21.15.46:0/3341296161 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3cc8003a60 con 0x7f3cd4105b60 2024-04-27T04:00:40.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.227+0000 7f3cd89da700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f3cbc06cce0 0x7f3cbc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:40.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.227+0000 7f3cd89da700 1 -- 172.21.15.46:0/3341296161 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3cc808d340 con 0x7f3cd4105b60 2024-04-27T04:00:40.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.227+0000 7f3cd2ffd700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f3cbc06cce0 0x7f3cbc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:40.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.227+0000 7f3cd89da700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853] conn(0x7f3cbc0729c0 0x7f3cbc074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:40.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.227+0000 7f3cd89da700 1 -- 172.21.15.46:0/3341296161 --> [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3cbc075490 con 0x7f3cbc0729c0 2024-04-27T04:00:40.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.227+0000 7f3cd89da700 1 -- 172.21.15.46:0/3341296161 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3cc808fa10 con 0x7f3cd4105b60 2024-04-27T04:00:40.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.227+0000 7f3cd37fe700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853] conn(0x7f3cbc0729c0 0x7f3cbc074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:40.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.227+0000 7f3cd2ffd700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f3cbc06cce0 0x7f3cbc06f1a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4005950 tx=0x7f3cc40058e0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:40.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.228+0000 7f3cd37fe700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853] conn(0x7f3cbc0729c0 0x7f3cbc074de0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:40.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.228+0000 7f3cd89da700 1 -- 172.21.15.46:0/3341296161 <== osd.0 v2:172.21.15.46:6802/2222489853 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f3cbc075490 con 0x7f3cbc0729c0 2024-04-27T04:00:40.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.244+0000 7f3cd99dc700 1 -- 172.21.15.46:0/3341296161 --> [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3cd40615e0 con 0x7f3cbc0729c0 2024-04-27T04:00:40.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.244+0000 7f3cd89da700 1 -- 172.21.15.46:0/3341296161 <== osd.0 v2:172.21.15.46:6802/2222489853 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3cd40615e0 con 0x7f3cbc0729c0 2024-04-27T04:00:40.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.244+0000 7f3cd99dc700 1 -- 172.21.15.46:0/3341296161 >> [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853] conn(0x7f3cbc0729c0 msgr2=0x7f3cbc074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:40.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.244+0000 7f3cd99dc700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853] conn(0x7f3cbc0729c0 0x7f3cbc074de0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:40.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.245+0000 7f3cd99dc700 1 -- 172.21.15.46:0/3341296161 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f3cbc06cce0 msgr2=0x7f3cbc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:40.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.245+0000 7f3cd99dc700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f3cbc06cce0 0x7f3cbc06f1a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4005950 tx=0x7f3cc40058e0 comp rx=0 tx=0).stop 2024-04-27T04:00:40.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.245+0000 7f3cd99dc700 1 -- 172.21.15.46:0/3341296161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd4105b60 msgr2=0x7f3cd4197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:40.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.245+0000 7f3cd99dc700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd4105b60 0x7f3cd4197e90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3cc800bd70 tx=0x7f3cc800be50 comp rx=0 tx=0).stop 2024-04-27T04:00:40.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.245+0000 7f3cd99dc700 1 -- 172.21.15.46:0/3341296161 shutdown_connections 2024-04-27T04:00:40.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.245+0000 7f3cd99dc700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.46:6802/2222489853,v1:172.21.15.46:6803/2222489853] conn(0x7f3cbc0729c0 0x7f3cbc074de0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:40.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.245+0000 7f3cd99dc700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f3cbc06cce0 0x7f3cbc06f1a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:40.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.245+0000 7f3cd99dc700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3cd40ffad0 0x7f3cd4197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:40.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.245+0000 7f3cd99dc700 1 --2- 172.21.15.46:0/3341296161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd4105b60 0x7f3cd4197e90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:40.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.245+0000 7f3cd99dc700 1 -- 172.21.15.46:0/3341296161 >> 172.21.15.46:0/3341296161 conn(0x7f3cd40fb5e0 msgr2=0x7f3cd40fc190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:40.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.245+0000 7f3cd99dc700 1 -- 172.21.15.46:0/3341296161 shutdown_connections 2024-04-27T04:00:40.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:40.245+0000 7f3cd99dc700 1 -- 172.21.15.46:0/3341296161 wait complete. 2024-04-27T04:00:40.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:40 smithi073 ceph-mon[34160]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:40.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:40 smithi046 ceph-mon[26899]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:40.971 INFO:teuthology.orchestra.run.smithi046.stdout:38654705701 2024-04-27T04:00:40.972 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-27T04:00:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:42.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.227+0000 7fe334952700 1 -- 172.21.15.46:0/3277995232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe330100e10 msgr2=0x7fe3301011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:42.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.227+0000 7fe334952700 1 --2- 172.21.15.46:0/3277995232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe330100e10 0x7fe3301011f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe320009910 tx=0x7fe320009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:42.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.228+0000 7fe334952700 1 -- 172.21.15.46:0/3277995232 shutdown_connections 2024-04-27T04:00:42.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.228+0000 7fe334952700 1 --2- 172.21.15.46:0/3277995232 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe3301017c0 0x7fe330105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:42.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.228+0000 7fe334952700 1 --2- 172.21.15.46:0/3277995232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe330100e10 0x7fe3301011f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:42.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.228+0000 7fe334952700 1 -- 172.21.15.46:0/3277995232 >> 172.21.15.46:0/3277995232 conn(0x7fe3300fc6e0 msgr2=0x7fe3300feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:42.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.228+0000 7fe334952700 1 -- 172.21.15.46:0/3277995232 shutdown_connections 2024-04-27T04:00:42.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.228+0000 7fe334952700 1 -- 172.21.15.46:0/3277995232 wait complete. 2024-04-27T04:00:42.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.229+0000 7fe334952700 1 Processor -- start 2024-04-27T04:00:42.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.229+0000 7fe334952700 1 -- start start 2024-04-27T04:00:42.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.229+0000 7fe334952700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe330100e10 0x7fe330196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:42.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.229+0000 7fe334952700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3301017c0 0x7fe3301970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:42.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.229+0000 7fe334952700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe330197790 con 0x7fe3301017c0 2024-04-27T04:00:42.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.229+0000 7fe334952700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33019b3a0 con 0x7fe330100e10 2024-04-27T04:00:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.229+0000 7fe32dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3301017c0 0x7fe3301970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.229+0000 7fe32dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3301017c0 0x7fe3301970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44300/0 (socket says 172.21.15.46:44300) 2024-04-27T04:00:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.229+0000 7fe32dd9b700 1 -- 172.21.15.46:0/587405635 learned_addr learned my addr 172.21.15.46:0/587405635 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.229+0000 7fe32e59c700 1 --2- 172.21.15.46:0/587405635 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe330100e10 0x7fe330196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.229+0000 7fe32dd9b700 1 -- 172.21.15.46:0/587405635 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe330100e10 msgr2=0x7fe330196b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.229+0000 7fe32dd9b700 1 --2- 172.21.15.46:0/587405635 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe330100e10 0x7fe330196b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.229+0000 7fe32dd9b700 1 -- 172.21.15.46:0/587405635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3200095f0 con 0x7fe3301017c0 2024-04-27T04:00:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.229+0000 7fe32dd9b700 1 --2- 172.21.15.46:0/587405635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3301017c0 0x7fe3301970b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe32400d900 tx=0x7fe32400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.230+0000 7fe31f7fe700 1 -- 172.21.15.46:0/587405635 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe324009900 con 0x7fe3301017c0 2024-04-27T04:00:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.230+0000 7fe31f7fe700 1 -- 172.21.15.46:0/587405635 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe324010460 con 0x7fe3301017c0 2024-04-27T04:00:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.230+0000 7fe31f7fe700 1 -- 172.21.15.46:0/587405635 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe32400f5f0 con 0x7fe3301017c0 2024-04-27T04:00:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.230+0000 7fe334952700 1 -- 172.21.15.46:0/587405635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33019b680 con 0x7fe3301017c0 2024-04-27T04:00:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.230+0000 7fe334952700 1 -- 172.21.15.46:0/587405635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33019bbd0 con 0x7fe3301017c0 2024-04-27T04:00:42.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.231+0000 7fe334952700 1 -- 172.21.15.46:0/587405635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3301091b0 con 0x7fe3301017c0 2024-04-27T04:00:42.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.231+0000 7fe31f7fe700 1 -- 172.21.15.46:0/587405635 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe3240105d0 con 0x7fe3301017c0 2024-04-27T04:00:42.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.231+0000 7fe31f7fe700 1 --2- 172.21.15.46:0/587405635 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe31806cce0 0x7fe31806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:42.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.232+0000 7fe31f7fe700 1 -- 172.21.15.46:0/587405635 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe32408b9b0 con 0x7fe3301017c0 2024-04-27T04:00:42.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.232+0000 7fe32e59c700 1 --2- 172.21.15.46:0/587405635 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe31806cce0 0x7fe31806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:42.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.232+0000 7fe32e59c700 1 --2- 172.21.15.46:0/587405635 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe31806cce0 0x7fe31806f1a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe320000c00 tx=0x7fe32001a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:42.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.236+0000 7fe31f7fe700 1 -- 172.21.15.46:0/587405635 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fe324056140 con 0x7fe3301017c0 2024-04-27T04:00:42.398 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:42 smithi046 ceph-mon[26899]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:42.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.443+0000 7fe334952700 1 -- 172.21.15.46:0/587405635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe330197dc0 con 0x7fe3301017c0 2024-04-27T04:00:42.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.443+0000 7fe31f7fe700 1 -- 172.21.15.46:0/587405635 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe324020090 con 0x7fe3301017c0 2024-04-27T04:00:42.444 INFO:teuthology.orchestra.run.smithi046.stdout:60129542176 2024-04-27T04:00:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.445+0000 7fe334952700 1 -- 172.21.15.46:0/587405635 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe31806cce0 msgr2=0x7fe31806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.445+0000 7fe334952700 1 --2- 172.21.15.46:0/587405635 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe31806cce0 0x7fe31806f1a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe320000c00 tx=0x7fe32001a040 comp rx=0 tx=0).stop 2024-04-27T04:00:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.445+0000 7fe334952700 1 -- 172.21.15.46:0/587405635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3301017c0 msgr2=0x7fe3301970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.445+0000 7fe334952700 1 --2- 172.21.15.46:0/587405635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3301017c0 0x7fe3301970b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe32400d900 tx=0x7fe32400dc10 comp rx=0 tx=0).stop 2024-04-27T04:00:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.445+0000 7fe334952700 1 -- 172.21.15.46:0/587405635 shutdown_connections 2024-04-27T04:00:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.445+0000 7fe334952700 1 --2- 172.21.15.46:0/587405635 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe31806cce0 0x7fe31806f1a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.445+0000 7fe334952700 1 --2- 172.21.15.46:0/587405635 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe330100e10 0x7fe330196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.445+0000 7fe334952700 1 --2- 172.21.15.46:0/587405635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3301017c0 0x7fe3301970b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.445+0000 7fe334952700 1 -- 172.21.15.46:0/587405635 >> 172.21.15.46:0/587405635 conn(0x7fe3300fc6e0 msgr2=0x7fe3300fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.445+0000 7fe334952700 1 -- 172.21.15.46:0/587405635 shutdown_connections 2024-04-27T04:00:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:42.445+0000 7fe334952700 1 -- 172.21.15.46:0/587405635 wait complete. 2024-04-27T04:00:42.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:42 smithi073 ceph-mon[34160]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:43.060 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:43.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.095+0000 7f93f5fa2700 1 -- 172.21.15.46:0/3378543500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00ff3b0 msgr2=0x7f93f00ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:43.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.095+0000 7f93f5fa2700 1 --2- 172.21.15.46:0/3378543500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00ff3b0 0x7f93f00ff790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f93e0009910 tx=0x7f93e0009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:43.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.095+0000 7f93f5fa2700 1 -- 172.21.15.46:0/3378543500 shutdown_connections 2024-04-27T04:00:43.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.095+0000 7f93f5fa2700 1 --2- 172.21.15.46:0/3378543500 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93f00ffd60 0x7f93f01058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:43.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.095+0000 7f93f5fa2700 1 --2- 172.21.15.46:0/3378543500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00ff3b0 0x7f93f00ff790 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.095+0000 7f93f5fa2700 1 -- 172.21.15.46:0/3378543500 >> 172.21.15.46:0/3378543500 conn(0x7f93f0075f40 msgr2=0x7f93f00fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.095+0000 7f93f5fa2700 1 -- 172.21.15.46:0/3378543500 shutdown_connections 2024-04-27T04:00:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.095+0000 7f93f5fa2700 1 -- 172.21.15.46:0/3378543500 wait complete. 2024-04-27T04:00:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.096+0000 7f93f5fa2700 1 Processor -- start 2024-04-27T04:00:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.096+0000 7f93f5fa2700 1 -- start start 2024-04-27T04:00:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.096+0000 7f93f5fa2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93f00ff3b0 0x7f93f0198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.096+0000 7f93f5fa2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00ffd60 0x7f93f0199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.096+0000 7f93f5fa2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f0199b20 con 0x7f93f00ffd60 2024-04-27T04:00:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.096+0000 7f93f5fa2700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f019d7f0 con 0x7f93f00ff3b0 2024-04-27T04:00:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.096+0000 7f93eeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00ffd60 0x7f93f0199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.096+0000 7f93eeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00ffd60 0x7f93f0199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44320/0 (socket says 172.21.15.46:44320) 2024-04-27T04:00:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.096+0000 7f93eeffd700 1 -- 172.21.15.46:0/2768022095 learned_addr learned my addr 172.21.15.46:0/2768022095 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.096+0000 7f93ef7fe700 1 --2- 172.21.15.46:0/2768022095 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93f00ff3b0 0x7f93f0198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.097+0000 7f93eeffd700 1 -- 172.21.15.46:0/2768022095 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93f00ff3b0 msgr2=0x7f93f0198f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.097+0000 7f93eeffd700 1 --2- 172.21.15.46:0/2768022095 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93f00ff3b0 0x7f93f0198f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.097+0000 7f93eeffd700 1 -- 172.21.15.46:0/2768022095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e00095f0 con 0x7f93f00ffd60 2024-04-27T04:00:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.097+0000 7f93ef7fe700 1 --2- 172.21.15.46:0/2768022095 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93f00ff3b0 0x7f93f0198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T04:00:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.097+0000 7f93eeffd700 1 --2- 172.21.15.46:0/2768022095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00ffd60 0x7f93f0199440 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f93e400b4d0 tx=0x7f93e400b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.097+0000 7f93ecff9700 1 -- 172.21.15.46:0/2768022095 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93e4011800 con 0x7f93f00ffd60 2024-04-27T04:00:43.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.097+0000 7f93ecff9700 1 -- 172.21.15.46:0/2768022095 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f93e4011e20 con 0x7f93f00ffd60 2024-04-27T04:00:43.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.097+0000 7f93ecff9700 1 -- 172.21.15.46:0/2768022095 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93e400f2e0 con 0x7f93f00ffd60 2024-04-27T04:00:43.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.097+0000 7f93f5fa2700 1 -- 172.21.15.46:0/2768022095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f019dad0 con 0x7f93f00ffd60 2024-04-27T04:00:43.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.097+0000 7f93f5fa2700 1 -- 172.21.15.46:0/2768022095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f019df70 con 0x7f93f00ffd60 2024-04-27T04:00:43.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.098+0000 7f93f5fa2700 1 -- 172.21.15.46:0/2768022095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f0102470 con 0x7f93f00ffd60 2024-04-27T04:00:43.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.099+0000 7f93ecff9700 1 -- 172.21.15.46:0/2768022095 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f93e4010380 con 0x7f93f00ffd60 2024-04-27T04:00:43.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.099+0000 7f93ecff9700 1 --2- 172.21.15.46:0/2768022095 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f93d806cce0 0x7f93d806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:43.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.099+0000 7f93ecff9700 1 -- 172.21.15.46:0/2768022095 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f93e408b970 con 0x7f93f00ffd60 2024-04-27T04:00:43.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.100+0000 7f93ef7fe700 1 --2- 172.21.15.46:0/2768022095 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f93d806cce0 0x7f93d806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:43.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.101+0000 7f93ef7fe700 1 --2- 172.21.15.46:0/2768022095 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f93d806cce0 0x7f93d806f1a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f93e0000c00 tx=0x7f93e000b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:43.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.105+0000 7f93ecff9700 1 -- 172.21.15.46:0/2768022095 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f93e4056080 con 0x7f93f00ffd60 2024-04-27T04:00:43.114 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:43.232 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:43 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/587405635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-27T04:00:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.317+0000 7f93f5fa2700 1 -- 172.21.15.46:0/2768022095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f93f004ebb0 con 0x7f93f00ffd60 2024-04-27T04:00:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.317+0000 7f93ecff9700 1 -- 172.21.15.46:0/2768022095 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f93e40596a0 con 0x7f93f00ffd60 2024-04-27T04:00:43.318 INFO:teuthology.orchestra.run.smithi046.stdout:81604378651 2024-04-27T04:00:43.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.320+0000 7f93f5fa2700 1 -- 172.21.15.46:0/2768022095 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f93d806cce0 msgr2=0x7f93d806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:43.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.320+0000 7f93f5fa2700 1 --2- 172.21.15.46:0/2768022095 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f93d806cce0 0x7f93d806f1a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f93e0000c00 tx=0x7f93e000b540 comp rx=0 tx=0).stop 2024-04-27T04:00:43.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.320+0000 7f93f5fa2700 1 -- 172.21.15.46:0/2768022095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00ffd60 msgr2=0x7f93f0199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:43.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.320+0000 7f93f5fa2700 1 --2- 172.21.15.46:0/2768022095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00ffd60 0x7f93f0199440 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f93e400b4d0 tx=0x7f93e400b7e0 comp rx=0 tx=0).stop 2024-04-27T04:00:43.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.320+0000 7f93f5fa2700 1 -- 172.21.15.46:0/2768022095 shutdown_connections 2024-04-27T04:00:43.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.320+0000 7f93f5fa2700 1 --2- 172.21.15.46:0/2768022095 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f93d806cce0 0x7f93d806f1a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:43.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.320+0000 7f93f5fa2700 1 --2- 172.21.15.46:0/2768022095 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93f00ff3b0 0x7f93f0198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:43.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.320+0000 7f93f5fa2700 1 --2- 172.21.15.46:0/2768022095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00ffd60 0x7f93f0199440 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:43.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.320+0000 7f93f5fa2700 1 -- 172.21.15.46:0/2768022095 >> 172.21.15.46:0/2768022095 conn(0x7f93f0075f40 msgr2=0x7f93f0104250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:43.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.320+0000 7f93f5fa2700 1 -- 172.21.15.46:0/2768022095 shutdown_connections 2024-04-27T04:00:43.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:43.320+0000 7f93f5fa2700 1 -- 172.21.15.46:0/2768022095 wait complete. 2024-04-27T04:00:43.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:43 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/587405635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-27T04:00:44.435 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:44.559 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:44 smithi046 ceph-mon[26899]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:44.559 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:44 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2768022095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-27T04:00:44.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:44 smithi073 ceph-mon[34160]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:44.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:44 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/2768022095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-27T04:00:45.379 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378651 for osd.2 2024-04-27T04:00:45.379 DEBUG:teuthology.parallel:result is None 2024-04-27T04:00:45.740 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542175 got 60129542176 for osd.1 2024-04-27T04:00:45.740 DEBUG:teuthology.parallel:result is None 2024-04-27T04:00:46.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:46 smithi046 ceph-mon[26899]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:46.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:46 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:00:46.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:46 smithi073 ceph-mon[34160]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:46.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:46 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:00:47.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.397+0000 7fbeababa700 1 -- 172.21.15.46:0/363886336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea40ffaf0 msgr2=0x7fbea40fff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:47.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.397+0000 7fbeababa700 1 --2- 172.21.15.46:0/363886336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea40ffaf0 0x7fbea40fff70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbe94009910 tx=0x7fbe94009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:47.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.397+0000 7fbeababa700 1 -- 172.21.15.46:0/363886336 shutdown_connections 2024-04-27T04:00:47.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.397+0000 7fbeababa700 1 --2- 172.21.15.46:0/363886336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea40ffaf0 0x7fbea40fff70 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:47.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.397+0000 7fbeababa700 1 --2- 172.21.15.46:0/363886336 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbea4105b80 0x7fbea4105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:47.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.397+0000 7fbeababa700 1 -- 172.21.15.46:0/363886336 >> 172.21.15.46:0/363886336 conn(0x7fbea40fb640 msgr2=0x7fbea40fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:47.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.397+0000 7fbeababa700 1 -- 172.21.15.46:0/363886336 shutdown_connections 2024-04-27T04:00:47.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.397+0000 7fbeababa700 1 -- 172.21.15.46:0/363886336 wait complete. 2024-04-27T04:00:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.398+0000 7fbeababa700 1 Processor -- start 2024-04-27T04:00:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.398+0000 7fbeababa700 1 -- start start 2024-04-27T04:00:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.398+0000 7fbeababa700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbea40ffaf0 0x7fbea4101750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.398+0000 7fbeababa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea4105b80 0x7fbea4101c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.398+0000 7fbeababa700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea41021d0 con 0x7fbea4105b80 2024-04-27T04:00:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.398+0000 7fbeababa700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea4102310 con 0x7fbea40ffaf0 2024-04-27T04:00:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.398+0000 7fbea9055700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea4105b80 0x7fbea4101c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.398+0000 7fbea9055700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea4105b80 0x7fbea4101c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44328/0 (socket says 172.21.15.46:44328) 2024-04-27T04:00:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.398+0000 7fbea9055700 1 -- 172.21.15.46:0/1335773892 learned_addr learned my addr 172.21.15.46:0/1335773892 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.398+0000 7fbea9856700 1 --2- 172.21.15.46:0/1335773892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbea40ffaf0 0x7fbea4101750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.398+0000 7fbea9055700 1 -- 172.21.15.46:0/1335773892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbea40ffaf0 msgr2=0x7fbea4101750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.398+0000 7fbea9055700 1 --2- 172.21.15.46:0/1335773892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbea40ffaf0 0x7fbea4101750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.398+0000 7fbea9055700 1 -- 172.21.15.46:0/1335773892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe940095f0 con 0x7fbea4105b80 2024-04-27T04:00:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.399+0000 7fbea9055700 1 --2- 172.21.15.46:0/1335773892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea4105b80 0x7fbea4101c90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbe94006010 tx=0x7fbe9400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.399+0000 7fbe9affd700 1 -- 172.21.15.46:0/1335773892 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe9401d070 con 0x7fbea4105b80 2024-04-27T04:00:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.399+0000 7fbe9affd700 1 -- 172.21.15.46:0/1335773892 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe9400f460 con 0x7fbea4105b80 2024-04-27T04:00:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.399+0000 7fbeababa700 1 -- 172.21.15.46:0/1335773892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea4102590 con 0x7fbea4105b80 2024-04-27T04:00:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.399+0000 7fbe9affd700 1 -- 172.21.15.46:0/1335773892 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe94017630 con 0x7fbea4105b80 2024-04-27T04:00:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.399+0000 7fbeababa700 1 -- 172.21.15.46:0/1335773892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea41a3050 con 0x7fbea4105b80 2024-04-27T04:00:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.400+0000 7fbeababa700 1 -- 172.21.15.46:0/1335773892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea41011f0 con 0x7fbea4105b80 2024-04-27T04:00:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.400+0000 7fbe9affd700 1 -- 172.21.15.46:0/1335773892 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbe9400f5d0 con 0x7fbea4105b80 2024-04-27T04:00:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.401+0000 7fbe9affd700 1 --2- 172.21.15.46:0/1335773892 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbe9006ccf0 0x7fbe9006f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.401+0000 7fbe9affd700 1 -- 172.21.15.46:0/1335773892 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fbe9408d2e0 con 0x7fbea4105b80 2024-04-27T04:00:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.401+0000 7fbea9856700 1 --2- 172.21.15.46:0/1335773892 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbe9006ccf0 0x7fbe9006f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.401+0000 7fbea9856700 1 --2- 172.21.15.46:0/1335773892 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbe9006ccf0 0x7fbe9006f1b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbea0009a10 tx=0x7fbea0009190 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:47.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.406+0000 7fbe9affd700 1 -- 172.21.15.46:0/1335773892 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fbe94057b30 con 0x7fbea4105b80 2024-04-27T04:00:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.612+0000 7fbeababa700 1 -- 172.21.15.46:0/1335773892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fbea410cd70 con 0x7fbea4105b80 2024-04-27T04:00:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.612+0000 7fbe9affd700 1 -- 172.21.15.46:0/1335773892 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fbe9405b150 con 0x7fbea4105b80 2024-04-27T04:00:47.613 INFO:teuthology.orchestra.run.smithi046.stdout:167503724556 2024-04-27T04:00:47.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.614+0000 7fbeababa700 1 -- 172.21.15.46:0/1335773892 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbe9006ccf0 msgr2=0x7fbe9006f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:47.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.614+0000 7fbeababa700 1 --2- 172.21.15.46:0/1335773892 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbe9006ccf0 0x7fbe9006f1b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbea0009a10 tx=0x7fbea0009190 comp rx=0 tx=0).stop 2024-04-27T04:00:47.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.614+0000 7fbeababa700 1 -- 172.21.15.46:0/1335773892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea4105b80 msgr2=0x7fbea4101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:47.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.614+0000 7fbeababa700 1 --2- 172.21.15.46:0/1335773892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea4105b80 0x7fbea4101c90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbe94006010 tx=0x7fbe9400bd70 comp rx=0 tx=0).stop 2024-04-27T04:00:47.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.614+0000 7fbeababa700 1 -- 172.21.15.46:0/1335773892 shutdown_connections 2024-04-27T04:00:47.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.614+0000 7fbeababa700 1 --2- 172.21.15.46:0/1335773892 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbe9006ccf0 0x7fbe9006f1b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:47.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.614+0000 7fbeababa700 1 --2- 172.21.15.46:0/1335773892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbea40ffaf0 0x7fbea4101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:47.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.614+0000 7fbeababa700 1 --2- 172.21.15.46:0/1335773892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea4105b80 0x7fbea4101c90 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:47.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.614+0000 7fbeababa700 1 -- 172.21.15.46:0/1335773892 >> 172.21.15.46:0/1335773892 conn(0x7fbea40fb640 msgr2=0x7fbea4103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:47.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.614+0000 7fbeababa700 1 -- 172.21.15.46:0/1335773892 shutdown_connections 2024-04-27T04:00:47.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:47.614+0000 7fbeababa700 1 -- 172.21.15.46:0/1335773892 wait complete. 2024-04-27T04:00:48.388 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:48 smithi046 ceph-mon[26899]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:48.388 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:48 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/1335773892' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-27T04:00:48.413 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724553 got 167503724556 for osd.5 2024-04-27T04:00:48.413 DEBUG:teuthology.parallel:result is None 2024-04-27T04:00:48.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:48 smithi073 ceph-mon[34160]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:48.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:48 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/1335773892' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-27T04:00:49.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.201+0000 7fc87a3bc700 1 -- 172.21.15.46:0/4247484245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8740ffab0 msgr2=0x7fc8740fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:49.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.201+0000 7fc87a3bc700 1 --2- 172.21.15.46:0/4247484245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8740ffab0 0x7fc8740fff30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc864009910 tx=0x7fc864009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:49.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.202+0000 7fc87a3bc700 1 -- 172.21.15.46:0/4247484245 shutdown_connections 2024-04-27T04:00:49.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.202+0000 7fc87a3bc700 1 --2- 172.21.15.46:0/4247484245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8740ffab0 0x7fc8740fff30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:49.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.202+0000 7fc87a3bc700 1 --2- 172.21.15.46:0/4247484245 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc874105ab0 0x7fc874105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:49.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.202+0000 7fc87a3bc700 1 -- 172.21.15.46:0/4247484245 >> 172.21.15.46:0/4247484245 conn(0x7fc8740fb5e0 msgr2=0x7fc8740fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:49.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.202+0000 7fc87a3bc700 1 -- 172.21.15.46:0/4247484245 shutdown_connections 2024-04-27T04:00:49.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.202+0000 7fc87a3bc700 1 -- 172.21.15.46:0/4247484245 wait complete. 2024-04-27T04:00:49.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.202+0000 7fc87a3bc700 1 Processor -- start 2024-04-27T04:00:49.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.202+0000 7fc87a3bc700 1 -- start start 2024-04-27T04:00:49.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc87a3bc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8740ffab0 0x7fc8740fbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:49.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc87a3bc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc874105ab0 0x7fc8740fc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:49.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc87a3bc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8740fc890 con 0x7fc874105ab0 2024-04-27T04:00:49.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc87a3bc700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8740fc9d0 con 0x7fc8740ffab0 2024-04-27T04:00:49.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc8737fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc874105ab0 0x7fc8740fc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:49.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc8737fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc874105ab0 0x7fc8740fc350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44368/0 (socket says 172.21.15.46:44368) 2024-04-27T04:00:49.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc8737fe700 1 -- 172.21.15.46:0/418193422 learned_addr learned my addr 172.21.15.46:0/418193422 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:49.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc873fff700 1 --2- 172.21.15.46:0/418193422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8740ffab0 0x7fc8740fbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:49.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc8737fe700 1 -- 172.21.15.46:0/418193422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8740ffab0 msgr2=0x7fc8740fbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:49.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc8737fe700 1 --2- 172.21.15.46:0/418193422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8740ffab0 0x7fc8740fbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:49.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc8737fe700 1 -- 172.21.15.46:0/418193422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8640095f0 con 0x7fc874105ab0 2024-04-27T04:00:49.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc873fff700 1 --2- 172.21.15.46:0/418193422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8740ffab0 0x7fc8740fbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T04:00:49.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc8737fe700 1 --2- 172.21.15.46:0/418193422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc874105ab0 0x7fc8740fc350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc864006010 tx=0x7fc86400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:49.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc8717fa700 1 -- 172.21.15.46:0/418193422 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc86401d070 con 0x7fc874105ab0 2024-04-27T04:00:49.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc8717fa700 1 -- 172.21.15.46:0/418193422 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8640056e0 con 0x7fc874105ab0 2024-04-27T04:00:49.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc87a3bc700 1 -- 172.21.15.46:0/418193422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8740fcc50 con 0x7fc874105ab0 2024-04-27T04:00:49.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.203+0000 7fc8717fa700 1 -- 172.21.15.46:0/418193422 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc864017400 con 0x7fc874105ab0 2024-04-27T04:00:49.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.204+0000 7fc87a3bc700 1 -- 172.21.15.46:0/418193422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8741a2ed0 con 0x7fc874105ab0 2024-04-27T04:00:49.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.204+0000 7fc87a3bc700 1 -- 172.21.15.46:0/418193422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8741090e0 con 0x7fc874105ab0 2024-04-27T04:00:49.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.205+0000 7fc8717fa700 1 -- 172.21.15.46:0/418193422 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc86400f6f0 con 0x7fc874105ab0 2024-04-27T04:00:49.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.205+0000 7fc8717fa700 1 --2- 172.21.15.46:0/418193422 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fc86006cce0 0x7fc86006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:49.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.206+0000 7fc8717fa700 1 -- 172.21.15.46:0/418193422 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc86408d2f0 con 0x7fc874105ab0 2024-04-27T04:00:49.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.206+0000 7fc873fff700 1 --2- 172.21.15.46:0/418193422 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fc86006cce0 0x7fc86006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:49.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.206+0000 7fc873fff700 1 --2- 172.21.15.46:0/418193422 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fc86006cce0 0x7fc86006f1a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc85c005fd0 tx=0x7fc85c005ee0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:49.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.210+0000 7fc8717fa700 1 -- 172.21.15.46:0/418193422 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fc864057a00 con 0x7fc874105ab0 2024-04-27T04:00:49.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.085+0000 7ff1b06a8700 1 -- 172.21.15.46:0/2864567450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a8100e20 msgr2=0x7ff1a8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:49.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.085+0000 7ff1b06a8700 1 --2- 172.21.15.46:0/2864567450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a8100e20 0x7ff1a8101200 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff19800b0a0 tx=0x7ff19800b3b0 comp rx=0 tx=0).stop 2024-04-27T04:00:49.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.085+0000 7ff1b06a8700 1 -- 172.21.15.46:0/2864567450 shutdown_connections 2024-04-27T04:00:49.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.085+0000 7ff1b06a8700 1 --2- 172.21.15.46:0/2864567450 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff1a81017d0 0x7ff1a8105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:49.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.085+0000 7ff1b06a8700 1 --2- 172.21.15.46:0/2864567450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a8100e20 0x7ff1a8101200 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:49.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.085+0000 7ff1b06a8700 1 -- 172.21.15.46:0/2864567450 >> 172.21.15.46:0/2864567450 conn(0x7ff1a80fc6d0 msgr2=0x7ff1a80feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:49.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.085+0000 7ff1b06a8700 1 -- 172.21.15.46:0/2864567450 shutdown_connections 2024-04-27T04:00:49.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.085+0000 7ff1b06a8700 1 -- 172.21.15.46:0/2864567450 wait complete. 2024-04-27T04:00:49.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.085+0000 7ff1b06a8700 1 Processor -- start 2024-04-27T04:00:49.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.085+0000 7ff1b06a8700 1 -- start start 2024-04-27T04:00:49.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1b06a8700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff1a8100e20 0x7ff1a819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:49.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1b06a8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a81017d0 0x7ff1a819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:49.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1b06a8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a819d3a0 con 0x7ff1a81017d0 2024-04-27T04:00:49.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1b06a8700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a81969d0 con 0x7ff1a8100e20 2024-04-27T04:00:49.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1adc43700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a81017d0 0x7ff1a819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:49.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1adc43700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a81017d0 0x7ff1a819cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44344/0 (socket says 172.21.15.46:44344) 2024-04-27T04:00:49.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1adc43700 1 -- 172.21.15.46:0/3941304694 learned_addr learned my addr 172.21.15.46:0/3941304694 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:49.313 INFO:teuthology.orchestra.run.smithi046.stdout:111669149718 2024-04-27T04:00:49.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1ae444700 1 --2- 172.21.15.46:0/3941304694 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff1a8100e20 0x7ff1a819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:49.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1adc43700 1 -- 172.21.15.46:0/3941304694 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff1a8100e20 msgr2=0x7ff1a819c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:49.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1adc43700 1 --2- 172.21.15.46:0/3941304694 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff1a8100e20 0x7ff1a819c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:49.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1adc43700 1 -- 172.21.15.46:0/3941304694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff198009d00 con 0x7ff1a81017d0 2024-04-27T04:00:49.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1adc43700 1 --2- 172.21.15.46:0/3941304694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a81017d0 0x7ff1a819cd80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00c8f0 tx=0x7ff19c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:49.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1a77fe700 1 -- 172.21.15.46:0/3941304694 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff19c007a50 con 0x7ff1a81017d0 2024-04-27T04:00:49.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1a77fe700 1 -- 172.21.15.46:0/3941304694 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff19c007bb0 con 0x7ff1a81017d0 2024-04-27T04:00:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1b06a8700 1 -- 172.21.15.46:0/3941304694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a8196cb0 con 0x7ff1a81017d0 2024-04-27T04:00:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1a77fe700 1 -- 172.21.15.46:0/3941304694 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff19c018630 con 0x7ff1a81017d0 2024-04-27T04:00:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.086+0000 7ff1b06a8700 1 -- 172.21.15.46:0/3941304694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a8197200 con 0x7ff1a81017d0 2024-04-27T04:00:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.087+0000 7ff1b06a8700 1 -- 172.21.15.46:0/3941304694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a81091c0 con 0x7ff1a81017d0 2024-04-27T04:00:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.088+0000 7ff1a77fe700 1 -- 172.21.15.46:0/3941304694 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff19c007d20 con 0x7ff1a81017d0 2024-04-27T04:00:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.088+0000 7ff1a77fe700 1 --2- 172.21.15.46:0/3941304694 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff19006cce0 0x7ff19006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.088+0000 7ff1a77fe700 1 -- 172.21.15.46:0/3941304694 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff19c08be10 con 0x7ff1a81017d0 2024-04-27T04:00:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.088+0000 7ff1ae444700 1 --2- 172.21.15.46:0/3941304694 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff19006cce0 0x7ff19006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.089+0000 7ff1ae444700 1 --2- 172.21.15.46:0/3941304694 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff19006cce0 0x7ff19006f1a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff198009850 tx=0x7ff1980097c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.093+0000 7ff1a77fe700 1 -- 172.21.15.46:0/3941304694 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7ff19c05a030 con 0x7ff1a81017d0 2024-04-27T04:00:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.304+0000 7ff1b06a8700 1 -- 172.21.15.46:0/3941304694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff1a8197e90 con 0x7ff1a81017d0 2024-04-27T04:00:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.305+0000 7ff1a77fe700 1 -- 172.21.15.46:0/3941304694 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff19c059bc0 con 0x7ff1a81017d0 2024-04-27T04:00:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.307+0000 7ff1b06a8700 1 -- 172.21.15.46:0/3941304694 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff19006cce0 msgr2=0x7ff19006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.307+0000 7ff1b06a8700 1 --2- 172.21.15.46:0/3941304694 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff19006cce0 0x7ff19006f1a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff198009850 tx=0x7ff1980097c0 comp rx=0 tx=0).stop 2024-04-27T04:00:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.307+0000 7ff1b06a8700 1 -- 172.21.15.46:0/3941304694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a81017d0 msgr2=0x7ff1a819cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.307+0000 7ff1b06a8700 1 --2- 172.21.15.46:0/3941304694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a81017d0 0x7ff1a819cd80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00c8f0 tx=0x7ff19c00ccb0 comp rx=0 tx=0).stop 2024-04-27T04:00:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.307+0000 7ff1b06a8700 1 -- 172.21.15.46:0/3941304694 shutdown_connections 2024-04-27T04:00:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.307+0000 7ff1b06a8700 1 --2- 172.21.15.46:0/3941304694 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff19006cce0 0x7ff19006f1a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.307+0000 7ff1b06a8700 1 --2- 172.21.15.46:0/3941304694 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff1a8100e20 0x7ff1a819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.307+0000 7ff1b06a8700 1 --2- 172.21.15.46:0/3941304694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a81017d0 0x7ff1a819cd80 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.307+0000 7ff1b06a8700 1 -- 172.21.15.46:0/3941304694 >> 172.21.15.46:0/3941304694 conn(0x7ff1a80fc6d0 msgr2=0x7ff1a80fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.307+0000 7ff1b06a8700 1 -- 172.21.15.46:0/3941304694 shutdown_connections 2024-04-27T04:00:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.307+0000 7ff1b06a8700 1 -- 172.21.15.46:0/3941304694 wait complete. 2024-04-27T04:00:49.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.416+0000 7fc87a3bc700 1 -- 172.21.15.46:0/418193422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fc8741011f0 con 0x7fc874105ab0 2024-04-27T04:00:49.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.417+0000 7fc8717fa700 1 -- 172.21.15.46:0/418193422 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc86405b020 con 0x7fc874105ab0 2024-04-27T04:00:49.418 INFO:teuthology.orchestra.run.smithi046.stdout:38654705703 2024-04-27T04:00:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.419+0000 7fc87a3bc700 1 -- 172.21.15.46:0/418193422 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fc86006cce0 msgr2=0x7fc86006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.419+0000 7fc87a3bc700 1 --2- 172.21.15.46:0/418193422 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fc86006cce0 0x7fc86006f1a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc85c005fd0 tx=0x7fc85c005ee0 comp rx=0 tx=0).stop 2024-04-27T04:00:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.419+0000 7fc87a3bc700 1 -- 172.21.15.46:0/418193422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc874105ab0 msgr2=0x7fc8740fc350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.419+0000 7fc87a3bc700 1 --2- 172.21.15.46:0/418193422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc874105ab0 0x7fc8740fc350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc864006010 tx=0x7fc86400be20 comp rx=0 tx=0).stop 2024-04-27T04:00:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.419+0000 7fc87a3bc700 1 -- 172.21.15.46:0/418193422 shutdown_connections 2024-04-27T04:00:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.419+0000 7fc87a3bc700 1 --2- 172.21.15.46:0/418193422 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fc86006cce0 0x7fc86006f1a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.419+0000 7fc87a3bc700 1 --2- 172.21.15.46:0/418193422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8740ffab0 0x7fc8740fbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.419+0000 7fc87a3bc700 1 --2- 172.21.15.46:0/418193422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc874105ab0 0x7fc8740fc350 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.419+0000 7fc87a3bc700 1 -- 172.21.15.46:0/418193422 >> 172.21.15.46:0/418193422 conn(0x7fc8740fb5e0 msgr2=0x7fc874069330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.419+0000 7fc87a3bc700 1 -- 172.21.15.46:0/418193422 shutdown_connections 2024-04-27T04:00:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:49.419+0000 7fc87a3bc700 1 -- 172.21.15.46:0/418193422 wait complete. 2024-04-27T04:00:49.930 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149716 got 111669149718 for osd.3 2024-04-27T04:00:49.930 DEBUG:teuthology.parallel:result is None 2024-04-27T04:00:50.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:50 smithi073 ceph-mon[34160]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:50.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:50 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/3941304694' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-27T04:00:50.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:50 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/418193422' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-27T04:00:50.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:50 smithi046 ceph-mon[26899]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:50.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:50 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/3941304694' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-27T04:00:50.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:50 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/418193422' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-27T04:00:51.105 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705701 got 38654705703 for osd.0 2024-04-27T04:00:51.106 DEBUG:teuthology.parallel:result is None 2024-04-27T04:00:51.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.568+0000 7f30e936f700 1 -- 172.21.15.46:0/3296281507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30e4101e00 msgr2=0x7f30e41021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:51.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.568+0000 7f30e936f700 1 --2- 172.21.15.46:0/3296281507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30e4101e00 0x7f30e41021e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f30d400b0a0 tx=0x7f30d400b3b0 comp rx=0 tx=0).stop 2024-04-27T04:00:51.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.568+0000 7f30e936f700 1 -- 172.21.15.46:0/3296281507 shutdown_connections 2024-04-27T04:00:51.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.568+0000 7f30e936f700 1 --2- 172.21.15.46:0/3296281507 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30e4102720 0x7f30e410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:51.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.568+0000 7f30e936f700 1 --2- 172.21.15.46:0/3296281507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30e4101e00 0x7f30e41021e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:51.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.568+0000 7f30e936f700 1 -- 172.21.15.46:0/3296281507 >> 172.21.15.46:0/3296281507 conn(0x7f30e4076740 msgr2=0x7f30e40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:51.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.568+0000 7f30e936f700 1 -- 172.21.15.46:0/3296281507 shutdown_connections 2024-04-27T04:00:51.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.568+0000 7f30e936f700 1 -- 172.21.15.46:0/3296281507 wait complete. 2024-04-27T04:00:51.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.569+0000 7f30e936f700 1 Processor -- start 2024-04-27T04:00:51.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.569+0000 7f30e936f700 1 -- start start 2024-04-27T04:00:51.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.569+0000 7f30e936f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30e4101e00 0x7f30e419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:51.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.569+0000 7f30e936f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30e4102720 0x7f30e419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:51.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.569+0000 7f30e936f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e419d3a0 con 0x7f30e4102720 2024-04-27T04:00:51.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.569+0000 7f30e936f700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e4196980 con 0x7f30e4101e00 2024-04-27T04:00:51.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.569+0000 7f30e27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30e4102720 0x7f30e419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:51.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.569+0000 7f30e27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30e4102720 0x7f30e419cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60896/0 (socket says 172.21.15.46:60896) 2024-04-27T04:00:51.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.569+0000 7f30e27fc700 1 -- 172.21.15.46:0/3454512509 learned_addr learned my addr 172.21.15.46:0/3454512509 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:51.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.569+0000 7f30e2ffd700 1 --2- 172.21.15.46:0/3454512509 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30e4101e00 0x7f30e419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:51.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.570+0000 7f30e27fc700 1 -- 172.21.15.46:0/3454512509 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30e4101e00 msgr2=0x7f30e419c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:51.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.570+0000 7f30e27fc700 1 --2- 172.21.15.46:0/3454512509 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30e4101e00 0x7f30e419c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:51.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.570+0000 7f30e27fc700 1 -- 172.21.15.46:0/3454512509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d4009d00 con 0x7f30e4102720 2024-04-27T04:00:51.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.570+0000 7f30e27fc700 1 --2- 172.21.15.46:0/3454512509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30e4102720 0x7f30e419cd80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f30d800c8f0 tx=0x7f30d800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:51.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.570+0000 7f30cbfff700 1 -- 172.21.15.46:0/3454512509 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30d80078a0 con 0x7f30e4102720 2024-04-27T04:00:51.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.570+0000 7f30e936f700 1 -- 172.21.15.46:0/3454512509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e4196c60 con 0x7f30e4102720 2024-04-27T04:00:51.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.570+0000 7f30cbfff700 1 -- 172.21.15.46:0/3454512509 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f30d800f450 con 0x7f30e4102720 2024-04-27T04:00:51.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.570+0000 7f30cbfff700 1 -- 172.21.15.46:0/3454512509 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30d8018610 con 0x7f30e4102720 2024-04-27T04:00:51.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.570+0000 7f30e936f700 1 -- 172.21.15.46:0/3454512509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e41971b0 con 0x7f30e4102720 2024-04-27T04:00:51.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.571+0000 7f30e936f700 1 -- 172.21.15.46:0/3454512509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e4105590 con 0x7f30e4102720 2024-04-27T04:00:51.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.572+0000 7f30cbfff700 1 -- 172.21.15.46:0/3454512509 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f30d8004830 con 0x7f30e4102720 2024-04-27T04:00:51.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.572+0000 7f30cbfff700 1 --2- 172.21.15.46:0/3454512509 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f30cc06ccf0 0x7f30cc06f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:51.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.572+0000 7f30cbfff700 1 -- 172.21.15.46:0/3454512509 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f30d808be50 con 0x7f30e4102720 2024-04-27T04:00:51.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.572+0000 7f30e2ffd700 1 --2- 172.21.15.46:0/3454512509 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f30cc06ccf0 0x7f30cc06f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:51.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.572+0000 7f30e2ffd700 1 --2- 172.21.15.46:0/3454512509 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f30cc06ccf0 0x7f30cc06f1b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f30d4000f80 tx=0x7f30d40097e0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:51.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.577+0000 7f30cbfff700 1 -- 172.21.15.46:0/3454512509 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f30d8059ff0 con 0x7f30e4102720 2024-04-27T04:00:51.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.781+0000 7f30e936f700 1 -- 172.21.15.46:0/3454512509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f30e4068e00 con 0x7f30e4102720 2024-04-27T04:00:51.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.781+0000 7f30cbfff700 1 -- 172.21.15.46:0/3454512509 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f30d8059b80 con 0x7f30e4102720 2024-04-27T04:00:51.782 INFO:teuthology.orchestra.run.smithi046.stdout:146028888082 2024-04-27T04:00:51.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.783+0000 7f30e936f700 1 -- 172.21.15.46:0/3454512509 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f30cc06ccf0 msgr2=0x7f30cc06f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:51.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.783+0000 7f30e936f700 1 --2- 172.21.15.46:0/3454512509 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f30cc06ccf0 0x7f30cc06f1b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f30d4000f80 tx=0x7f30d40097e0 comp rx=0 tx=0).stop 2024-04-27T04:00:51.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.783+0000 7f30e936f700 1 -- 172.21.15.46:0/3454512509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30e4102720 msgr2=0x7f30e419cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:51.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.783+0000 7f30e936f700 1 --2- 172.21.15.46:0/3454512509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30e4102720 0x7f30e419cd80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f30d800c8f0 tx=0x7f30d800cc00 comp rx=0 tx=0).stop 2024-04-27T04:00:51.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.783+0000 7f30e936f700 1 -- 172.21.15.46:0/3454512509 shutdown_connections 2024-04-27T04:00:51.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.784+0000 7f30e936f700 1 --2- 172.21.15.46:0/3454512509 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f30cc06ccf0 0x7f30cc06f1b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:51.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.784+0000 7f30e936f700 1 --2- 172.21.15.46:0/3454512509 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30e4101e00 0x7f30e419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:51.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.784+0000 7f30e936f700 1 --2- 172.21.15.46:0/3454512509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30e4102720 0x7f30e419cd80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:51.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.784+0000 7f30e936f700 1 -- 172.21.15.46:0/3454512509 >> 172.21.15.46:0/3454512509 conn(0x7f30e4076740 msgr2=0x7f30e41030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:51.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.784+0000 7f30e936f700 1 -- 172.21.15.46:0/3454512509 shutdown_connections 2024-04-27T04:00:51.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:51.784+0000 7f30e936f700 1 -- 172.21.15.46:0/3454512509 wait complete. 2024-04-27T04:00:52.438 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888078 got 146028888082 for osd.4 2024-04-27T04:00:52.438 DEBUG:teuthology.parallel:result is None 2024-04-27T04:00:52.439 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-27T04:00:52.439 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-27T04:00:52.464 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:52 smithi046 ceph-mon[26899]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:52.464 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:52 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/3454512509' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-27T04:00:52.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:52 smithi073 ceph-mon[34160]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:52.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:52 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/3454512509' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-27T04:00:52.656 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:54.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.232+0000 7f5380bd9700 1 -- 172.21.15.46:0/3894807379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f537c105b60 msgr2=0x7f537c105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:54.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.232+0000 7f5380bd9700 1 --2- 172.21.15.46:0/3894807379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f537c105b60 0x7f537c105f40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f536c009910 tx=0x7f536c009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:54.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.232+0000 7f5380bd9700 1 -- 172.21.15.46:0/3894807379 shutdown_connections 2024-04-27T04:00:54.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.232+0000 7f5380bd9700 1 --2- 172.21.15.46:0/3894807379 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f537c0ffad0 0x7f537c0fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:54.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.232+0000 7f5380bd9700 1 --2- 172.21.15.46:0/3894807379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f537c105b60 0x7f537c105f40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:54.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.232+0000 7f5380bd9700 1 -- 172.21.15.46:0/3894807379 >> 172.21.15.46:0/3894807379 conn(0x7f537c0fb5e0 msgr2=0x7f537c0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:54.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.232+0000 7f5380bd9700 1 -- 172.21.15.46:0/3894807379 shutdown_connections 2024-04-27T04:00:54.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.232+0000 7f5380bd9700 1 -- 172.21.15.46:0/3894807379 wait complete. 2024-04-27T04:00:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.233+0000 7f5380bd9700 1 Processor -- start 2024-04-27T04:00:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.233+0000 7f5380bd9700 1 -- start start 2024-04-27T04:00:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.233+0000 7f5380bd9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f537c0ffad0 0x7f537c19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.233+0000 7f5380bd9700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f537c105b60 0x7f537c19ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.233+0000 7f5380bd9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537c19d450 con 0x7f537c0ffad0 2024-04-27T04:00:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.233+0000 7f5380bd9700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537c196a30 con 0x7f537c105b60 2024-04-27T04:00:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.234+0000 7f537a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f537c0ffad0 0x7f537c19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.234+0000 7f537a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f537c0ffad0 0x7f537c19c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60912/0 (socket says 172.21.15.46:60912) 2024-04-27T04:00:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.234+0000 7f537a59c700 1 -- 172.21.15.46:0/4119761364 learned_addr learned my addr 172.21.15.46:0/4119761364 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.234+0000 7f5379d9b700 1 --2- 172.21.15.46:0/4119761364 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f537c105b60 0x7f537c19ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.234+0000 7f537a59c700 1 -- 172.21.15.46:0/4119761364 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f537c105b60 msgr2=0x7f537c19ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.234+0000 7f537a59c700 1 --2- 172.21.15.46:0/4119761364 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f537c105b60 0x7f537c19ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.234+0000 7f537a59c700 1 -- 172.21.15.46:0/4119761364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536c0095f0 con 0x7f537c0ffad0 2024-04-27T04:00:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.234+0000 7f5379d9b700 1 --2- 172.21.15.46:0/4119761364 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f537c105b60 0x7f537c19ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T04:00:54.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.234+0000 7f537a59c700 1 --2- 172.21.15.46:0/4119761364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f537c0ffad0 0x7f537c19c8f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f536c00bd70 tx=0x7f536c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:54.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.234+0000 7f536b7fe700 1 -- 172.21.15.46:0/4119761364 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f536c01d070 con 0x7f537c0ffad0 2024-04-27T04:00:54.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.234+0000 7f5380bd9700 1 -- 172.21.15.46:0/4119761364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f537c196cb0 con 0x7f537c0ffad0 2024-04-27T04:00:54.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.234+0000 7f536b7fe700 1 -- 172.21.15.46:0/4119761364 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f536c00f460 con 0x7f537c0ffad0 2024-04-27T04:00:54.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.234+0000 7f536b7fe700 1 -- 172.21.15.46:0/4119761364 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f536c017630 con 0x7f537c0ffad0 2024-04-27T04:00:54.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.234+0000 7f5380bd9700 1 -- 172.21.15.46:0/4119761364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f537c1971a0 con 0x7f537c0ffad0 2024-04-27T04:00:54.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.236+0000 7f5380bd9700 1 -- 172.21.15.46:0/4119761364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f537c1090e0 con 0x7f537c0ffad0 2024-04-27T04:00:54.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.236+0000 7f536b7fe700 1 -- 172.21.15.46:0/4119761364 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f536c00f5d0 con 0x7f537c0ffad0 2024-04-27T04:00:54.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.236+0000 7f536b7fe700 1 --2- 172.21.15.46:0/4119761364 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f536406cce0 0x7f536406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:54.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.236+0000 7f536b7fe700 1 -- 172.21.15.46:0/4119761364 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f536c08d490 con 0x7f537c0ffad0 2024-04-27T04:00:54.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.236+0000 7f5379d9b700 1 --2- 172.21.15.46:0/4119761364 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f536406cce0 0x7f536406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:54.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.237+0000 7f5379d9b700 1 --2- 172.21.15.46:0/4119761364 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f536406cce0 0x7f536406f1a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f537c197bf0 tx=0x7f5370008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:54.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.243+0000 7f536b7fe700 1 -- 172.21.15.46:0/4119761364 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f536c05b630 con 0x7f537c0ffad0 2024-04-27T04:00:54.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:54 smithi046 ceph-mon[26899]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:54.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.446+0000 7f5380bd9700 1 -- 172.21.15.46:0/4119761364 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f537c10cd50 con 0x7f536406cce0 2024-04-27T04:00:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.447+0000 7f536b7fe700 1 -- 172.21.15.46:0/4119761364 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28142 (secure 0 0 0) 0x7f537c10cd50 con 0x7f536406cce0 2024-04-27T04:00:54.448 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:00:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.449+0000 7f5380bd9700 1 -- 172.21.15.46:0/4119761364 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f536406cce0 msgr2=0x7f536406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.449+0000 7f5380bd9700 1 --2- 172.21.15.46:0/4119761364 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f536406cce0 0x7f536406f1a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f537c197bf0 tx=0x7f5370008040 comp rx=0 tx=0).stop 2024-04-27T04:00:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.449+0000 7f5380bd9700 1 -- 172.21.15.46:0/4119761364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f537c0ffad0 msgr2=0x7f537c19c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.449+0000 7f5380bd9700 1 --2- 172.21.15.46:0/4119761364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f537c0ffad0 0x7f537c19c8f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f536c00bd70 tx=0x7f536c00be50 comp rx=0 tx=0).stop 2024-04-27T04:00:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.449+0000 7f5380bd9700 1 -- 172.21.15.46:0/4119761364 shutdown_connections 2024-04-27T04:00:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.449+0000 7f5380bd9700 1 --2- 172.21.15.46:0/4119761364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f537c0ffad0 0x7f537c19c8f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.449+0000 7f5380bd9700 1 --2- 172.21.15.46:0/4119761364 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f536406cce0 0x7f536406f1a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.449+0000 7f5380bd9700 1 --2- 172.21.15.46:0/4119761364 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f537c105b60 0x7f537c19ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.449+0000 7f5380bd9700 1 -- 172.21.15.46:0/4119761364 >> 172.21.15.46:0/4119761364 conn(0x7f537c0fb5e0 msgr2=0x7f537c104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.450+0000 7f5380bd9700 1 -- 172.21.15.46:0/4119761364 shutdown_connections 2024-04-27T04:00:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:54.450+0000 7f5380bd9700 1 -- 172.21.15.46:0/4119761364 wait complete. 2024-04-27T04:00:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-27T04:00:54.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:54 smithi073 ceph-mon[34160]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:55.083 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-04-27T04:00:52.764604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":182388,"kb_used_data":3876,"kb_used_omap":9,"kb_used_meta":178422,"kb_avail":562288524,"statfs":{"total":575970213888,"available":575783448576,"internally_reserved":0,"allocated":3969024,"data_stored":2609166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":182704806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002277"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":186,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-27T04:00:02.975563+0000","last_change":"2024-04-27T03:59:35.513954+0000","last_active":"2024-04-27T04:00:02.975563+0000","last_peered":"2024-04-27T04:00:02.975563+0000","last_clean":"2024-04-27T04:00:02.975563+0000","last_became_active":"2024-04-27T03:59:35.513613+0000","last_became_peered":"2024-04-27T03:59:35.513613+0000","last_unstale":"2024-04-27T04:00:02.975563+0000","last_undegraded":"2024-04-27T04:00:02.975563+0000","last_fullsized":"2024-04-27T04:00:02.975563+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:58:45.624847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:58:45.624847+0000","last_clean_scrub_stamp":"2024-04-27T03:58:45.624847+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:16:18.228796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31472,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713680,"statfs":{"total":95995035648,"available":95962808320,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31476,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713676,"statfs":{"total":95995035648,"available":95962804224,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.47999999999999998}]},{"osd":1,"last update":"Sat Apr 27 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.55500000000000005}]},{"osd":2,"last update":"Sat Apr 27 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.45200000000000001}]},{"osd":3,"last update":"Sat Apr 27 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.42499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 04:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.46700000000000003}]},{"osd":1,"last update":"Sat Apr 27 04:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.443}]},{"osd":2,"last update":"Sat Apr 27 04:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.54000000000000004}]},{"osd":4,"last update":"Sat Apr 27 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.36099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717776,"statfs":{"total":95995035648,"available":95967002624,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.36199999999999999,"15min":0.36799999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.45900000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.217},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.36599999999999999,"15min":0.372},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.48099999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.25900000000000001}]},{"osd":1,"last update":"Sat Apr 27 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.40300000000000002,"15min":0.41099999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.51500000000000001,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.245},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.40999999999999998,"15min":0.41899999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55200000000000005,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.28899999999999998}]},{"osd":3,"last update":"Sat Apr 27 04:00:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.40999999999999998}]},{"osd":4,"last update":"Sat Apr 27 04:00:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.39000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36148,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93709004,"statfs":{"total":95995035648,"available":95958020096,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 27 04:00:24 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.33700000000000002,"15min":0.33300000000000002},"min":{"1min":0.251,"5min":0.189,"15min":0.189},"max":{"1min":0.47399999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.34399999999999997,"15min":0.34100000000000003},"min":{"1min":0.28799999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.47299999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.28999999999999998}]},{"osd":2,"last update":"Sat Apr 27 04:00:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.375,"15min":0.38300000000000001},"min":{"1min":0.25,"5min":0.249,"15min":0.249},"max":{"1min":0.439,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.39500000000000002,"15min":0.40100000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.48699999999999999,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.30399999999999999}]},{"osd":3,"last update":"Sat Apr 27 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.298,"15min":1.298},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":13.542999999999999,"5min":13.542999999999999,"15min":13.542999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.71199999999999997}]},{"osd":4,"last update":"Sat Apr 27 04:00:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.745}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 04:00:21 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.33500000000000002,"15min":0.32900000000000001},"min":{"1min":0.28699999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.505,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.33900000000000002,"15min":0.33500000000000002},"min":{"1min":0.27800000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54500000000000004,"5min":0.626,"15min":0.626},"last":0.377}]},{"osd":2,"last update":"Sat Apr 27 03:59:46 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"last":0.41499999999999998}]},{"osd":3,"last update":"Sat Apr 27 04:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.629}]},{"osd":4,"last update":"Sat Apr 27 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.61499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T04:00:55.084 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-27T04:00:55.340 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:55.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:55 smithi046 ceph-mon[26899]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T04:00:55.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:55 smithi073 ceph-mon[34160]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T04:00:56.474 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:56 smithi046 ceph-mon[26899]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:56.567 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:56 smithi073 ceph-mon[34160]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:57.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.034+0000 7f9fae183700 1 -- 172.21.15.46:0/3884852874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa80732b0 msgr2=0x7f9fa8073690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:57.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.034+0000 7f9fae183700 1 --2- 172.21.15.46:0/3884852874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa80732b0 0x7f9fa8073690 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9f98009910 tx=0x7f9f98009c20 comp rx=0 tx=0).stop 2024-04-27T04:00:57.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.034+0000 7f9fae183700 1 -- 172.21.15.46:0/3884852874 shutdown_connections 2024-04-27T04:00:57.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.034+0000 7f9fae183700 1 --2- 172.21.15.46:0/3884852874 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9fa8073bd0 0x7f9fa810cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:57.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.034+0000 7f9fae183700 1 --2- 172.21.15.46:0/3884852874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa80732b0 0x7f9fa8073690 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:57.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.034+0000 7f9fae183700 1 -- 172.21.15.46:0/3884852874 >> 172.21.15.46:0/3884852874 conn(0x7f9fa80fc6d0 msgr2=0x7f9fa80feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:57.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.034+0000 7f9fae183700 1 -- 172.21.15.46:0/3884852874 shutdown_connections 2024-04-27T04:00:57.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.034+0000 7f9fae183700 1 -- 172.21.15.46:0/3884852874 wait complete. 2024-04-27T04:00:57.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.035+0000 7f9fae183700 1 Processor -- start 2024-04-27T04:00:57.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.035+0000 7f9fae183700 1 -- start start 2024-04-27T04:00:57.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.035+0000 7f9fae183700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa80732b0 0x7f9fa819e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:57.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.035+0000 7f9fae183700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9fa8073bd0 0x7f9fa819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:57.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.035+0000 7f9fae183700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa819f550 con 0x7f9fa80732b0 2024-04-27T04:00:57.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.035+0000 7f9fae183700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa8198b30 con 0x7f9fa8073bd0 2024-04-27T04:00:57.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.036+0000 7f9fa77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa80732b0 0x7f9fa819e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:57.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.036+0000 7f9fa77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa80732b0 0x7f9fa819e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60932/0 (socket says 172.21.15.46:60932) 2024-04-27T04:00:57.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.036+0000 7f9fa77fe700 1 -- 172.21.15.46:0/584662849 learned_addr learned my addr 172.21.15.46:0/584662849 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:57.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.036+0000 7f9fa6ffd700 1 --2- 172.21.15.46:0/584662849 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9fa8073bd0 0x7f9fa819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:57.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.036+0000 7f9fa77fe700 1 -- 172.21.15.46:0/584662849 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9fa8073bd0 msgr2=0x7f9fa819ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:57.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.036+0000 7f9fa77fe700 1 --2- 172.21.15.46:0/584662849 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9fa8073bd0 0x7f9fa819ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:57.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.036+0000 7f9fa77fe700 1 -- 172.21.15.46:0/584662849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f980095f0 con 0x7f9fa80732b0 2024-04-27T04:00:57.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.036+0000 7f9fa77fe700 1 --2- 172.21.15.46:0/584662849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa80732b0 0x7f9fa819e9f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800bd70 tx=0x7f9f9800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:57.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.036+0000 7f9fa4ff9700 1 -- 172.21.15.46:0/584662849 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f9801d070 con 0x7f9fa80732b0 2024-04-27T04:00:57.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.036+0000 7f9fae183700 1 -- 172.21.15.46:0/584662849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa8198db0 con 0x7f9fa80732b0 2024-04-27T04:00:57.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.036+0000 7f9fa4ff9700 1 -- 172.21.15.46:0/584662849 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f980056e0 con 0x7f9fa80732b0 2024-04-27T04:00:57.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.036+0000 7f9fa4ff9700 1 -- 172.21.15.46:0/584662849 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f98017400 con 0x7f9fa80732b0 2024-04-27T04:00:57.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.036+0000 7f9fae183700 1 -- 172.21.15.46:0/584662849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa81992a0 con 0x7f9fa80732b0 2024-04-27T04:00:57.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.037+0000 7f9fae183700 1 -- 172.21.15.46:0/584662849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa8107830 con 0x7f9fa80732b0 2024-04-27T04:00:57.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.038+0000 7f9fa4ff9700 1 -- 172.21.15.46:0/584662849 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9f98003a60 con 0x7f9fa80732b0 2024-04-27T04:00:57.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.038+0000 7f9fa4ff9700 1 --2- 172.21.15.46:0/584662849 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9f9006cce0 0x7f9f9006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:57.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.038+0000 7f9fa4ff9700 1 -- 172.21.15.46:0/584662849 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f9f9808d2a0 con 0x7f9fa80732b0 2024-04-27T04:00:57.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.038+0000 7f9fa6ffd700 1 --2- 172.21.15.46:0/584662849 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9f9006cce0 0x7f9f9006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:57.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.039+0000 7f9fa6ffd700 1 --2- 172.21.15.46:0/584662849 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9f9006cce0 0x7f9f9006f1a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c005950 tx=0x7f9f9c0058e0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:57.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.043+0000 7f9fa4ff9700 1 -- 172.21.15.46:0/584662849 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f9f9805b4c0 con 0x7f9fa80732b0 2024-04-27T04:00:57.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.240+0000 7f9fae183700 1 -- 172.21.15.46:0/584662849 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9fa819a060 con 0x7f9f9006cce0 2024-04-27T04:00:57.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.242+0000 7f9fa4ff9700 1 -- 172.21.15.46:0/584662849 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28171 (secure 0 0 0) 0x7f9fa819a060 con 0x7f9f9006cce0 2024-04-27T04:00:57.243 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:00:57.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.244+0000 7f9fae183700 1 -- 172.21.15.46:0/584662849 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9f9006cce0 msgr2=0x7f9f9006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:57.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.244+0000 7f9fae183700 1 --2- 172.21.15.46:0/584662849 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9f9006cce0 0x7f9f9006f1a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c005950 tx=0x7f9f9c0058e0 comp rx=0 tx=0).stop 2024-04-27T04:00:57.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.244+0000 7f9fae183700 1 -- 172.21.15.46:0/584662849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa80732b0 msgr2=0x7f9fa819e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:57.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.244+0000 7f9fae183700 1 --2- 172.21.15.46:0/584662849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa80732b0 0x7f9fa819e9f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800bd70 tx=0x7f9f9800be50 comp rx=0 tx=0).stop 2024-04-27T04:00:57.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.244+0000 7f9fae183700 1 -- 172.21.15.46:0/584662849 shutdown_connections 2024-04-27T04:00:57.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.244+0000 7f9fae183700 1 --2- 172.21.15.46:0/584662849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa80732b0 0x7f9fa819e9f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:57.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.244+0000 7f9fae183700 1 --2- 172.21.15.46:0/584662849 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9f9006cce0 0x7f9f9006f1a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:57.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.244+0000 7f9fae183700 1 --2- 172.21.15.46:0/584662849 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9fa8073bd0 0x7f9fa819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:57.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.244+0000 7f9fae183700 1 -- 172.21.15.46:0/584662849 >> 172.21.15.46:0/584662849 conn(0x7f9fa80fc6d0 msgr2=0x7f9fa81053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:57.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.244+0000 7f9fae183700 1 -- 172.21.15.46:0/584662849 shutdown_connections 2024-04-27T04:00:57.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:57.244+0000 7f9fae183700 1 -- 172.21.15.46:0/584662849 wait complete. 2024-04-27T04:00:57.246 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-27T04:00:57.827 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-04-27T04:00:56.765375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":182388,"kb_used_data":3876,"kb_used_omap":9,"kb_used_meta":178422,"kb_avail":562288524,"statfs":{"total":575970213888,"available":575783448576,"internally_reserved":0,"allocated":3969024,"data_stored":2609166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":182704806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002258"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":186,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-27T04:00:02.975563+0000","last_change":"2024-04-27T03:59:35.513954+0000","last_active":"2024-04-27T04:00:02.975563+0000","last_peered":"2024-04-27T04:00:02.975563+0000","last_clean":"2024-04-27T04:00:02.975563+0000","last_became_active":"2024-04-27T03:59:35.513613+0000","last_became_peered":"2024-04-27T03:59:35.513613+0000","last_unstale":"2024-04-27T04:00:02.975563+0000","last_undegraded":"2024-04-27T04:00:02.975563+0000","last_fullsized":"2024-04-27T04:00:02.975563+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:58:45.624847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:58:45.624847+0000","last_clean_scrub_stamp":"2024-04-27T03:58:45.624847+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:16:18.228796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31472,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713680,"statfs":{"total":95995035648,"available":95962808320,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31476,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713676,"statfs":{"total":95995035648,"available":95962804224,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.51100000000000001}]},{"osd":1,"last update":"Sat Apr 27 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.58999999999999997}]},{"osd":2,"last update":"Sat Apr 27 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.45300000000000001}]},{"osd":3,"last update":"Sat Apr 27 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 04:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.41899999999999998}]},{"osd":1,"last update":"Sat Apr 27 04:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.501},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.47499999999999998}]},{"osd":2,"last update":"Sat Apr 27 04:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.48499999999999999}]},{"osd":4,"last update":"Sat Apr 27 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.31}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378654,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717776,"statfs":{"total":95995035648,"available":95967002624,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.36199999999999999,"15min":0.36799999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.45900000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.36599999999999999,"15min":0.372},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.48099999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.34000000000000002}]},{"osd":1,"last update":"Sat Apr 27 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.40300000000000002,"15min":0.41099999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.51500000000000001,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.40999999999999998,"15min":0.41899999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55200000000000005,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.36699999999999999}]},{"osd":3,"last update":"Sat Apr 27 04:00:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.58199999999999996}]},{"osd":4,"last update":"Sat Apr 27 04:00:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.53000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36148,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93709004,"statfs":{"total":95995035648,"available":95958020096,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 27 04:00:24 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.33700000000000002,"15min":0.33300000000000002},"min":{"1min":0.251,"5min":0.189,"15min":0.189},"max":{"1min":0.47399999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.251},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.34399999999999997,"15min":0.34100000000000003},"min":{"1min":0.28799999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.47299999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.28699999999999998}]},{"osd":2,"last update":"Sat Apr 27 04:00:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.375,"15min":0.38300000000000001},"min":{"1min":0.25,"5min":0.249,"15min":0.249},"max":{"1min":0.439,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.39500000000000002,"15min":0.40100000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.48699999999999999,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.30299999999999999}]},{"osd":3,"last update":"Sat Apr 27 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.298,"15min":1.298},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":13.542999999999999,"5min":13.542999999999999,"15min":13.542999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.50900000000000001}]},{"osd":4,"last update":"Sat Apr 27 04:00:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.47999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 04:00:21 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.33500000000000002,"15min":0.32900000000000001},"min":{"1min":0.28699999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.505,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.33900000000000002,"15min":0.33500000000000002},"min":{"1min":0.27800000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54500000000000004,"5min":0.626,"15min":0.626},"last":0.28100000000000003}]},{"osd":2,"last update":"Sat Apr 27 04:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.38300000000000001,"15min":0.39100000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.46100000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.247},{"interface":"front","average":{"1min":0.371,"5min":0.44700000000000001,"15min":0.46000000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.503,"5min":1.5620000000000001,"15min":1.5620000000000001},"last":0.30099999999999999}]},{"osd":3,"last update":"Sat Apr 27 04:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.40500000000000003}]},{"osd":4,"last update":"Sat Apr 27 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.42499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T04:00:57.828 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-27T04:00:57.829 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-27T04:00:57.829 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-27T04:00:57.829 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-27T04:00:58.037 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:00:58.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:58 smithi046 ceph-mon[26899]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:58.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:00:58 smithi046 ceph-mon[26899]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T04:00:58.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:58 smithi073 ceph-mon[34160]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:00:58.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:00:58 smithi073 ceph-mon[34160]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T04:00:59.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.542+0000 7f84c55ae700 1 -- 172.21.15.46:0/2379833830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c0100420 msgr2=0x7f84c0100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:59.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.542+0000 7f84c55ae700 1 --2- 172.21.15.46:0/2379833830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c0100420 0x7f84c0100800 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f84b000b0a0 tx=0x7f84b000b3b0 comp rx=0 tx=0).stop 2024-04-27T04:00:59.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.542+0000 7f84c55ae700 1 -- 172.21.15.46:0/2379833830 shutdown_connections 2024-04-27T04:00:59.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.542+0000 7f84c55ae700 1 --2- 172.21.15.46:0/2379833830 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84c0100dd0 0x7f84c01047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:59.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.542+0000 7f84c55ae700 1 --2- 172.21.15.46:0/2379833830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c0100420 0x7f84c0100800 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:59.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.542+0000 7f84c55ae700 1 -- 172.21.15.46:0/2379833830 >> 172.21.15.46:0/2379833830 conn(0x7f84c0075f40 msgr2=0x7f84c00fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:59.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.542+0000 7f84c55ae700 1 -- 172.21.15.46:0/2379833830 shutdown_connections 2024-04-27T04:00:59.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.542+0000 7f84c55ae700 1 -- 172.21.15.46:0/2379833830 wait complete. 2024-04-27T04:00:59.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.542+0000 7f84c55ae700 1 Processor -- start 2024-04-27T04:00:59.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84c55ae700 1 -- start start 2024-04-27T04:00:59.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84c55ae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c0100420 0x7f84c019ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84c55ae700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84c0100dd0 0x7f84c019cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84c55ae700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c0196ab0 con 0x7f84c0100420 2024-04-27T04:00:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84c55ae700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c0196c20 con 0x7f84c0100dd0 2024-04-27T04:00:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84beffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c0100420 0x7f84c019ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84beffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c0100420 0x7f84c019ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60942/0 (socket says 172.21.15.46:60942) 2024-04-27T04:00:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84beffd700 1 -- 172.21.15.46:0/2908032164 learned_addr learned my addr 172.21.15.46:0/2908032164 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:00:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84be7fc700 1 --2- 172.21.15.46:0/2908032164 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84c0100dd0 0x7f84c019cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84beffd700 1 -- 172.21.15.46:0/2908032164 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84c0100dd0 msgr2=0x7f84c019cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84beffd700 1 --2- 172.21.15.46:0/2908032164 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84c0100dd0 0x7f84c019cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84beffd700 1 -- 172.21.15.46:0/2908032164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b0009d00 con 0x7f84c0100420 2024-04-27T04:00:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84beffd700 1 --2- 172.21.15.46:0/2908032164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c0100420 0x7f84c019ca00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f84b0000f80 tx=0x7f84b0007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84b7fff700 1 -- 172.21.15.46:0/2908032164 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84b000e040 con 0x7f84c0100420 2024-04-27T04:00:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84c55ae700 1 -- 172.21.15.46:0/2908032164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c0196ea0 con 0x7f84c0100420 2024-04-27T04:00:59.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84b7fff700 1 -- 172.21.15.46:0/2908032164 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84b0003de0 con 0x7f84c0100420 2024-04-27T04:00:59.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84b7fff700 1 -- 172.21.15.46:0/2908032164 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84b001d7d0 con 0x7f84c0100420 2024-04-27T04:00:59.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.543+0000 7f84c55ae700 1 -- 172.21.15.46:0/2908032164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c0197390 con 0x7f84c0100420 2024-04-27T04:00:59.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.545+0000 7f84c55ae700 1 -- 172.21.15.46:0/2908032164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c01013e0 con 0x7f84c0100420 2024-04-27T04:00:59.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.545+0000 7f84b7fff700 1 -- 172.21.15.46:0/2908032164 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f84b0004040 con 0x7f84c0100420 2024-04-27T04:00:59.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.545+0000 7f84b7fff700 1 --2- 172.21.15.46:0/2908032164 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f84ac06ccf0 0x7f84ac06f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:00:59.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.545+0000 7f84b7fff700 1 -- 172.21.15.46:0/2908032164 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f84b008d850 con 0x7f84c0100420 2024-04-27T04:00:59.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.545+0000 7f84be7fc700 1 --2- 172.21.15.46:0/2908032164 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f84ac06ccf0 0x7f84ac06f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:00:59.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.546+0000 7f84be7fc700 1 --2- 172.21.15.46:0/2908032164 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f84ac06ccf0 0x7f84ac06f1b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f84a8006fd0 tx=0x7f84a8008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:00:59.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.550+0000 7f84b7fff700 1 -- 172.21.15.46:0/2908032164 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f84b0057f60 con 0x7f84c0100420 2024-04-27T04:00:59.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.797+0000 7f84c55ae700 1 -- 172.21.15.46:0/2908032164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f84c0197f90 con 0x7f84c0100420 2024-04-27T04:00:59.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.798+0000 7f84b7fff700 1 -- 172.21.15.46:0/2908032164 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f84b005b580 con 0x7f84c0100420 2024-04-27T04:00:59.801 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:00:59.801 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-27T04:00:59.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.800+0000 7f84c55ae700 1 -- 172.21.15.46:0/2908032164 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f84ac06ccf0 msgr2=0x7f84ac06f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:59.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.800+0000 7f84c55ae700 1 --2- 172.21.15.46:0/2908032164 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f84ac06ccf0 0x7f84ac06f1b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f84a8006fd0 tx=0x7f84a8008040 comp rx=0 tx=0).stop 2024-04-27T04:00:59.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.800+0000 7f84c55ae700 1 -- 172.21.15.46:0/2908032164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c0100420 msgr2=0x7f84c019ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:00:59.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.800+0000 7f84c55ae700 1 --2- 172.21.15.46:0/2908032164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c0100420 0x7f84c019ca00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f84b0000f80 tx=0x7f84b0007bd0 comp rx=0 tx=0).stop 2024-04-27T04:00:59.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.800+0000 7f84c55ae700 1 -- 172.21.15.46:0/2908032164 shutdown_connections 2024-04-27T04:00:59.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.800+0000 7f84c55ae700 1 --2- 172.21.15.46:0/2908032164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c0100420 0x7f84c019ca00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:59.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.800+0000 7f84c55ae700 1 --2- 172.21.15.46:0/2908032164 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f84ac06ccf0 0x7f84ac06f1b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:59.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.800+0000 7f84c55ae700 1 --2- 172.21.15.46:0/2908032164 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84c0100dd0 0x7f84c019cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:00:59.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.800+0000 7f84c55ae700 1 -- 172.21.15.46:0/2908032164 >> 172.21.15.46:0/2908032164 conn(0x7f84c0075f40 msgr2=0x7f84c00fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:00:59.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.801+0000 7f84c55ae700 1 -- 172.21.15.46:0/2908032164 shutdown_connections 2024-04-27T04:00:59.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:00:59.801+0000 7f84c55ae700 1 -- 172.21.15.46:0/2908032164 wait complete. 2024-04-27T04:01:00.511 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:00 smithi046 ceph-mon[26899]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:00.512 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:00 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2908032164' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-27T04:01:00.537 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-27T04:01:00.537 INFO:tasks.cephadm:Setup complete, yielding 2024-04-27T04:01:00.537 INFO:teuthology.run_tasks:Running task print... 2024-04-27T04:01:00.545 INFO:teuthology.task.print:**** done end installing reef cephadm ... 2024-04-27T04:01:00.545 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T04:01:00.555 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi046.front.sepia.ceph.com 2024-04-27T04:01:00.555 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-27T04:01:00.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:00 smithi073 ceph-mon[34160]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:00.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:00 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/2908032164' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-27T04:01:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:01:01.542 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:01 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:01:01.562 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:01 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:01:02.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:02 smithi073 ceph-mon[34160]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:02.389 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:02 smithi046 ceph-mon[26899]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:02.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.408+0000 7f694bb2e700 1 -- 172.21.15.46:0/1357638564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6944100c40 msgr2=0x7f6944101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:02.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.408+0000 7f694bb2e700 1 --2- 172.21.15.46:0/1357638564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6944100c40 0x7f6944101060 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6940009910 tx=0x7f6940009c20 comp rx=0 tx=0).stop 2024-04-27T04:01:02.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.408+0000 7f694bb2e700 1 -- 172.21.15.46:0/1357638564 shutdown_connections 2024-04-27T04:01:02.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.408+0000 7f694bb2e700 1 --2- 172.21.15.46:0/1357638564 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6944101e40 0x7f69441022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:02.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.408+0000 7f694bb2e700 1 --2- 172.21.15.46:0/1357638564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6944100c40 0x7f6944101060 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:02.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.408+0000 7f694bb2e700 1 -- 172.21.15.46:0/1357638564 >> 172.21.15.46:0/1357638564 conn(0x7f69440fc200 msgr2=0x7f69440fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:02.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.408+0000 7f694bb2e700 1 -- 172.21.15.46:0/1357638564 shutdown_connections 2024-04-27T04:01:02.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.408+0000 7f694bb2e700 1 -- 172.21.15.46:0/1357638564 wait complete. 2024-04-27T04:01:02.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.409+0000 7f694bb2e700 1 Processor -- start 2024-04-27T04:01:02.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.409+0000 7f694bb2e700 1 -- start start 2024-04-27T04:01:02.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.409+0000 7f694bb2e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6944100c40 0x7f6944074c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:02.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.409+0000 7f694bb2e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6944101e40 0x7f69440732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:02.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.409+0000 7f694bb2e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6944073810 con 0x7f6944100c40 2024-04-27T04:01:02.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.409+0000 7f694bb2e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6944073950 con 0x7f6944101e40 2024-04-27T04:01:02.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.409+0000 7f69498ca700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6944100c40 0x7f6944074c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:02.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.409+0000 7f69498ca700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6944100c40 0x7f6944074c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36764/0 (socket says 172.21.15.46:36764) 2024-04-27T04:01:02.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.409+0000 7f69498ca700 1 -- 172.21.15.46:0/42024152 learned_addr learned my addr 172.21.15.46:0/42024152 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:02.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.409+0000 7f69490c9700 1 --2- 172.21.15.46:0/42024152 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6944101e40 0x7f69440732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:02.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.410+0000 7f69498ca700 1 -- 172.21.15.46:0/42024152 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6944101e40 msgr2=0x7f69440732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:02.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.410+0000 7f69498ca700 1 --2- 172.21.15.46:0/42024152 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6944101e40 0x7f69440732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:02.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.410+0000 7f69498ca700 1 -- 172.21.15.46:0/42024152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69400095f0 con 0x7f6944100c40 2024-04-27T04:01:02.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.410+0000 7f69498ca700 1 --2- 172.21.15.46:0/42024152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6944100c40 0x7f6944074c30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f69400059d0 tx=0x7f694000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:02.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.410+0000 7f693affd700 1 -- 172.21.15.46:0/42024152 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f694001d070 con 0x7f6944100c40 2024-04-27T04:01:02.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.410+0000 7f693affd700 1 -- 172.21.15.46:0/42024152 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6940005dc0 con 0x7f6944100c40 2024-04-27T04:01:02.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.410+0000 7f694bb2e700 1 -- 172.21.15.46:0/42024152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6944073bd0 con 0x7f6944100c40 2024-04-27T04:01:02.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.410+0000 7f693affd700 1 -- 172.21.15.46:0/42024152 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6940021410 con 0x7f6944100c40 2024-04-27T04:01:02.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.410+0000 7f694bb2e700 1 -- 172.21.15.46:0/42024152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6944074140 con 0x7f6944100c40 2024-04-27T04:01:02.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.411+0000 7f694bb2e700 1 -- 172.21.15.46:0/42024152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69441907c0 con 0x7f6944100c40 2024-04-27T04:01:02.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.411+0000 7f693affd700 1 -- 172.21.15.46:0/42024152 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f694000f460 con 0x7f6944100c40 2024-04-27T04:01:02.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.412+0000 7f693affd700 1 --2- 172.21.15.46:0/42024152 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f693006cce0 0x7f693006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:02.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.412+0000 7f693affd700 1 -- 172.21.15.46:0/42024152 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f694008cf40 con 0x7f6944100c40 2024-04-27T04:01:02.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.412+0000 7f69490c9700 1 --2- 172.21.15.46:0/42024152 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f693006cce0 0x7f693006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:02.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.412+0000 7f69490c9700 1 --2- 172.21.15.46:0/42024152 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f693006cce0 0x7f693006f1a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6934005950 tx=0x7f693400b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:02.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.416+0000 7f693affd700 1 -- 172.21.15.46:0/42024152 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f69400c0c50 con 0x7f6944100c40 2024-04-27T04:01:02.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.618+0000 7f694bb2e700 1 -- 172.21.15.46:0/42024152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f6944109a50 con 0x7f6944100c40 2024-04-27T04:01:02.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.622+0000 7f693affd700 1 -- 172.21.15.46:0/42024152 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v23)=0 v23) v1 ==== 143+0+0 (secure 0 0 0) 0x7f694005b0e0 con 0x7f6944100c40 2024-04-27T04:01:02.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.624+0000 7f694bb2e700 1 -- 172.21.15.46:0/42024152 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f693006cce0 msgr2=0x7f693006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:02.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.624+0000 7f694bb2e700 1 --2- 172.21.15.46:0/42024152 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f693006cce0 0x7f693006f1a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6934005950 tx=0x7f693400b410 comp rx=0 tx=0).stop 2024-04-27T04:01:02.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.625+0000 7f694bb2e700 1 -- 172.21.15.46:0/42024152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6944100c40 msgr2=0x7f6944074c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:02.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.625+0000 7f694bb2e700 1 --2- 172.21.15.46:0/42024152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6944100c40 0x7f6944074c30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f69400059d0 tx=0x7f694000bd70 comp rx=0 tx=0).stop 2024-04-27T04:01:02.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.625+0000 7f694bb2e700 1 -- 172.21.15.46:0/42024152 shutdown_connections 2024-04-27T04:01:02.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.625+0000 7f694bb2e700 1 --2- 172.21.15.46:0/42024152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6944100c40 0x7f6944074c30 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:02.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.625+0000 7f694bb2e700 1 --2- 172.21.15.46:0/42024152 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f693006cce0 0x7f693006f1a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:02.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.625+0000 7f694bb2e700 1 --2- 172.21.15.46:0/42024152 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6944101e40 0x7f69440732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:02.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.625+0000 7f694bb2e700 1 -- 172.21.15.46:0/42024152 >> 172.21.15.46:0/42024152 conn(0x7f69440fc200 msgr2=0x7f6944105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:02.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.625+0000 7f694bb2e700 1 -- 172.21.15.46:0/42024152 shutdown_connections 2024-04-27T04:01:02.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:02.625+0000 7f694bb2e700 1 -- 172.21.15.46:0/42024152 wait complete. 2024-04-27T04:01:03.491 INFO:teuthology.run_tasks:Running task print... 2024-04-27T04:01:03.500 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-27T04:01:03.501 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T04:01:03.509 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi046.front.sepia.ceph.com 2024-04-27T04:01:03.510 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-27T04:01:03.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:03 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/42024152' entity='client.admin' 2024-04-27T04:01:03.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:01:03.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:01:03.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:01:03.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:03 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:03.729 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:01:04.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:03 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/42024152' entity='client.admin' 2024-04-27T04:01:04.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:03 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:01:04.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:03 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:01:04.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:03 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:01:04.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:03 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:04.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:04 smithi046 ceph-mon[26899]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:05.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:04 smithi073 ceph-mon[34160]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.711+0000 7fbfe59bd700 1 -- 172.21.15.46:0/1350051646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfe0100c40 msgr2=0x7fbfe0101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.711+0000 7fbfe59bd700 1 --2- 172.21.15.46:0/1350051646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfe0100c40 0x7fbfe0101060 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009910 tx=0x7fbfd0009c20 comp rx=0 tx=0).stop 2024-04-27T04:01:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.711+0000 7fbfe59bd700 1 -- 172.21.15.46:0/1350051646 shutdown_connections 2024-04-27T04:01:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.711+0000 7fbfe59bd700 1 --2- 172.21.15.46:0/1350051646 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbfe0101e40 0x7fbfe01022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.711+0000 7fbfe59bd700 1 --2- 172.21.15.46:0/1350051646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfe0100c40 0x7fbfe0101060 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.711+0000 7fbfe59bd700 1 -- 172.21.15.46:0/1350051646 >> 172.21.15.46:0/1350051646 conn(0x7fbfe00fc200 msgr2=0x7fbfe00fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.711+0000 7fbfe59bd700 1 -- 172.21.15.46:0/1350051646 shutdown_connections 2024-04-27T04:01:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.711+0000 7fbfe59bd700 1 -- 172.21.15.46:0/1350051646 wait complete. 2024-04-27T04:01:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.712+0000 7fbfe59bd700 1 Processor -- start 2024-04-27T04:01:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.712+0000 7fbfe59bd700 1 -- start start 2024-04-27T04:01:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.712+0000 7fbfe59bd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfe0100c40 0x7fbfe0074be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:05.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.712+0000 7fbfe59bd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbfe0101e40 0x7fbfe00732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:05.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.712+0000 7fbfe59bd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe0073810 con 0x7fbfe0100c40 2024-04-27T04:01:05.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.712+0000 7fbfe59bd700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe0073950 con 0x7fbfe0101e40 2024-04-27T04:01:05.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.712+0000 7fbfdeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfe0100c40 0x7fbfe0074be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:05.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.712+0000 7fbfdeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfe0100c40 0x7fbfe0074be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36782/0 (socket says 172.21.15.46:36782) 2024-04-27T04:01:05.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.712+0000 7fbfdeffd700 1 -- 172.21.15.46:0/3054007390 learned_addr learned my addr 172.21.15.46:0/3054007390 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:05.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.713+0000 7fbfdeffd700 1 -- 172.21.15.46:0/3054007390 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbfe0101e40 msgr2=0x7fbfe00732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:05.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.713+0000 7fbfde7fc700 1 --2- 172.21.15.46:0/3054007390 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbfe0101e40 0x7fbfe00732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:05.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.713+0000 7fbfdeffd700 1 --2- 172.21.15.46:0/3054007390 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbfe0101e40 0x7fbfe00732d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:05.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.713+0000 7fbfdeffd700 1 -- 172.21.15.46:0/3054007390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd00095f0 con 0x7fbfe0100c40 2024-04-27T04:01:05.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.713+0000 7fbfdeffd700 1 --2- 172.21.15.46:0/3054007390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfe0100c40 0x7fbfe0074be0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00059d0 tx=0x7fbfd000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:05.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.713+0000 7fbfe49bb700 1 -- 172.21.15.46:0/3054007390 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd001d070 con 0x7fbfe0100c40 2024-04-27T04:01:05.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.713+0000 7fbfe49bb700 1 -- 172.21.15.46:0/3054007390 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbfd000f460 con 0x7fbfe0100c40 2024-04-27T04:01:05.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.713+0000 7fbfe59bd700 1 -- 172.21.15.46:0/3054007390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe0073bd0 con 0x7fbfe0100c40 2024-04-27T04:01:05.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.713+0000 7fbfe49bb700 1 -- 172.21.15.46:0/3054007390 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd0021640 con 0x7fbfe0100c40 2024-04-27T04:01:05.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.713+0000 7fbfe59bd700 1 -- 172.21.15.46:0/3054007390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe0074140 con 0x7fbfe0100c40 2024-04-27T04:01:05.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.714+0000 7fbfe59bd700 1 -- 172.21.15.46:0/3054007390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe0190840 con 0x7fbfe0100c40 2024-04-27T04:01:05.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.715+0000 7fbfe49bb700 1 -- 172.21.15.46:0/3054007390 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbfd000f5d0 con 0x7fbfe0100c40 2024-04-27T04:01:05.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.715+0000 7fbfe49bb700 1 --2- 172.21.15.46:0/3054007390 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbfc806cce0 0x7fbfc806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:05.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.715+0000 7fbfe49bb700 1 -- 172.21.15.46:0/3054007390 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fbfd008d170 con 0x7fbfe0100c40 2024-04-27T04:01:05.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.715+0000 7fbfde7fc700 1 --2- 172.21.15.46:0/3054007390 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbfc806cce0 0x7fbfc806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:05.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.716+0000 7fbfde7fc700 1 --2- 172.21.15.46:0/3054007390 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbfc806cce0 0x7fbfc806f1a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4006fd0 tx=0x7fbfd4008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:05.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.720+0000 7fbfe49bb700 1 -- 172.21.15.46:0/3054007390 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fbfd005b310 con 0x7fbfe0100c40 2024-04-27T04:01:05.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.942+0000 7fbfe59bd700 1 -- 172.21.15.46:0/3054007390 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbfe0074140 con 0x7fbfc806cce0 2024-04-27T04:01:05.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.943+0000 7fbfe49bb700 1 -- 172.21.15.46:0/3054007390 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fbfe0074140 con 0x7fbfc806cce0 2024-04-27T04:01:05.944 INFO:teuthology.orchestra.run.smithi046.stdout:Backend: cephadm 2024-04-27T04:01:05.944 INFO:teuthology.orchestra.run.smithi046.stdout:Available: Yes 2024-04-27T04:01:05.944 INFO:teuthology.orchestra.run.smithi046.stdout:Paused: No 2024-04-27T04:01:05.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.945+0000 7fbfe59bd700 1 -- 172.21.15.46:0/3054007390 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbfc806cce0 msgr2=0x7fbfc806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:05.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.945+0000 7fbfe59bd700 1 --2- 172.21.15.46:0/3054007390 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbfc806cce0 0x7fbfc806f1a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4006fd0 tx=0x7fbfd4008040 comp rx=0 tx=0).stop 2024-04-27T04:01:05.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.945+0000 7fbfe59bd700 1 -- 172.21.15.46:0/3054007390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfe0100c40 msgr2=0x7fbfe0074be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:05.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.945+0000 7fbfe59bd700 1 --2- 172.21.15.46:0/3054007390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfe0100c40 0x7fbfe0074be0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00059d0 tx=0x7fbfd000be20 comp rx=0 tx=0).stop 2024-04-27T04:01:05.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.945+0000 7fbfe59bd700 1 -- 172.21.15.46:0/3054007390 shutdown_connections 2024-04-27T04:01:05.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.945+0000 7fbfe59bd700 1 --2- 172.21.15.46:0/3054007390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfe0100c40 0x7fbfe0074be0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:05.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.945+0000 7fbfe59bd700 1 --2- 172.21.15.46:0/3054007390 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fbfc806cce0 0x7fbfc806f1a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:05.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.945+0000 7fbfe59bd700 1 --2- 172.21.15.46:0/3054007390 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbfe0101e40 0x7fbfe00732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:05.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.945+0000 7fbfe59bd700 1 -- 172.21.15.46:0/3054007390 >> 172.21.15.46:0/3054007390 conn(0x7fbfe00fc200 msgr2=0x7fbfe0105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:05.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.945+0000 7fbfe59bd700 1 -- 172.21.15.46:0/3054007390 shutdown_connections 2024-04-27T04:01:05.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:05.945+0000 7fbfe59bd700 1 -- 172.21.15.46:0/3054007390 wait complete. 2024-04-27T04:01:06.592 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-27T04:01:06.811 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:01:06.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:06 smithi046 ceph-mon[26899]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:07.047 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:06 smithi073 ceph-mon[34160]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:07.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:07 smithi046 ceph-mon[26899]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:01:08.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:07 smithi073 ceph-mon[34160]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:01:08.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.381+0000 7f44c141c700 1 -- 172.21.15.46:0/1266187612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44bc101e30 msgr2=0x7f44bc102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:08.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.381+0000 7f44c141c700 1 --2- 172.21.15.46:0/1266187612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44bc101e30 0x7f44bc102290 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f44b0009910 tx=0x7f44b0009c20 comp rx=0 tx=0).stop 2024-04-27T04:01:08.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.381+0000 7f44c141c700 1 -- 172.21.15.46:0/1266187612 shutdown_connections 2024-04-27T04:01:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.381+0000 7f44c141c700 1 --2- 172.21.15.46:0/1266187612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44bc101e30 0x7f44bc102290 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.381+0000 7f44c141c700 1 --2- 172.21.15.46:0/1266187612 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44bc100c30 0x7f44bc101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.381+0000 7f44c141c700 1 -- 172.21.15.46:0/1266187612 >> 172.21.15.46:0/1266187612 conn(0x7f44bc0fc1d0 msgr2=0x7f44bc0fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.381+0000 7f44c141c700 1 -- 172.21.15.46:0/1266187612 shutdown_connections 2024-04-27T04:01:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.382+0000 7f44c141c700 1 -- 172.21.15.46:0/1266187612 wait complete. 2024-04-27T04:01:08.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.382+0000 7f44c141c700 1 Processor -- start 2024-04-27T04:01:08.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.382+0000 7f44c141c700 1 -- start start 2024-04-27T04:01:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.383+0000 7f44c141c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44bc100c30 0x7f44bc196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.383+0000 7f44c141c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44bc101e30 0x7f44bc196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.383+0000 7f44c141c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc197110 con 0x7f44bc100c30 2024-04-27T04:01:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.383+0000 7f44c141c700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc197250 con 0x7f44bc101e30 2024-04-27T04:01:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.383+0000 7f44baffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44bc100c30 0x7f44bc196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.383+0000 7f44baffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44bc100c30 0x7f44bc196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36798/0 (socket says 172.21.15.46:36798) 2024-04-27T04:01:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.383+0000 7f44baffd700 1 -- 172.21.15.46:0/4179315027 learned_addr learned my addr 172.21.15.46:0/4179315027 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.383+0000 7f44ba7fc700 1 --2- 172.21.15.46:0/4179315027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44bc101e30 0x7f44bc196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.383+0000 7f44baffd700 1 -- 172.21.15.46:0/4179315027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44bc101e30 msgr2=0x7f44bc196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.383+0000 7f44baffd700 1 --2- 172.21.15.46:0/4179315027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44bc101e30 0x7f44bc196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.383+0000 7f44baffd700 1 -- 172.21.15.46:0/4179315027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b00095f0 con 0x7f44bc100c30 2024-04-27T04:01:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.383+0000 7f44baffd700 1 --2- 172.21.15.46:0/4179315027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44bc100c30 0x7f44bc196640 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f44ac00c910 tx=0x7f44ac00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.383+0000 7f44a3fff700 1 -- 172.21.15.46:0/4179315027 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44ac0041f0 con 0x7f44bc100c30 2024-04-27T04:01:08.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.383+0000 7f44c141c700 1 -- 172.21.15.46:0/4179315027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44bc19bc70 con 0x7f44bc100c30 2024-04-27T04:01:08.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.383+0000 7f44a3fff700 1 -- 172.21.15.46:0/4179315027 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44ac007dc0 con 0x7f44bc100c30 2024-04-27T04:01:08.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.383+0000 7f44a3fff700 1 -- 172.21.15.46:0/4179315027 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44ac003e70 con 0x7f44bc100c30 2024-04-27T04:01:08.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.384+0000 7f44c141c700 1 -- 172.21.15.46:0/4179315027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44bc19c190 con 0x7f44bc100c30 2024-04-27T04:01:08.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.384+0000 7f44c141c700 1 -- 172.21.15.46:0/4179315027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44bc1907b0 con 0x7f44bc100c30 2024-04-27T04:01:08.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.385+0000 7f44a3fff700 1 -- 172.21.15.46:0/4179315027 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f44ac004350 con 0x7f44bc100c30 2024-04-27T04:01:08.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.385+0000 7f44a3fff700 1 --2- 172.21.15.46:0/4179315027 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f44a406cc90 0x7f44a406f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:08.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.386+0000 7f44a3fff700 1 -- 172.21.15.46:0/4179315027 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f44ac08bc90 con 0x7f44bc100c30 2024-04-27T04:01:08.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.386+0000 7f44ba7fc700 1 --2- 172.21.15.46:0/4179315027 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f44a406cc90 0x7f44a406f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:08.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.386+0000 7f44ba7fc700 1 --2- 172.21.15.46:0/4179315027 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f44a406cc90 0x7f44a406f150 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f44b00051c0 tx=0x7f44b000b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.390+0000 7f44a3fff700 1 -- 172.21.15.46:0/4179315027 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f44ac059e30 con 0x7f44bc100c30 2024-04-27T04:01:08.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.601+0000 7f44c141c700 1 -- 172.21.15.46:0/4179315027 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f44bc02ceb0 con 0x7f44a406cc90 2024-04-27T04:01:08.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.611+0000 7f44a3fff700 1 -- 172.21.15.46:0/4179315027 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3100 (secure 0 0 0) 0x7f44bc02ceb0 con 0x7f44a406cc90 2024-04-27T04:01:08.611 INFO:teuthology.orchestra.run.smithi046.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T04:01:08.612 INFO:teuthology.orchestra.run.smithi046.stdout:alertmanager.smithi046 smithi046 *:9093,9094 running (3m) 2m ago 5m 25.9M - 0.25.0 c8568f914cd2 698cffee765e 2024-04-27T04:01:08.612 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-exporter.smithi046 smithi046 running (5m) 2m ago 5m 7059k - 18.2.2-1260-g5403a021 267a473d505d 6eec9c927d3f 2024-04-27T04:01:08.612 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-exporter.smithi073 smithi073 running (4m) 67s ago 4m 7004k - 18.2.2-1260-g5403a021 267a473d505d d101eef4888d 2024-04-27T04:01:08.612 INFO:teuthology.orchestra.run.smithi046.stdout:crash.smithi046 smithi046 running (5m) 2m ago 5m 6660k - 18.2.2-1260-g5403a021 267a473d505d 338c449faabd 2024-04-27T04:01:08.612 INFO:teuthology.orchestra.run.smithi046.stdout:crash.smithi073 smithi073 running (4m) 67s ago 4m 6660k - 18.2.2-1260-g5403a021 267a473d505d be1aaee1c655 2024-04-27T04:01:08.612 INFO:teuthology.orchestra.run.smithi046.stdout:grafana.smithi046 smithi046 *:3000 running (3m) 2m ago 5m 78.2M - 9.4.7 954c08fa6188 a834a3988304 2024-04-27T04:01:08.612 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.smithi046.qrfdvr smithi046 *:9283,8765,8443 running (6m) 2m ago 6m 491M - 18.2.2-1260-g5403a021 267a473d505d ba1d99ba4a26 2024-04-27T04:01:08.612 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.smithi073.awyllt smithi073 *:8443,9283,8765 running (4m) 67s ago 4m 451M - 18.2.2-1260-g5403a021 267a473d505d 06769682a51f 2024-04-27T04:01:08.612 INFO:teuthology.orchestra.run.smithi046.stdout:mon.smithi046 smithi046 running (6m) 2m ago 7m 43.3M 2048M 18.2.2-1260-g5403a021 267a473d505d 72fac097effc 2024-04-27T04:01:08.612 INFO:teuthology.orchestra.run.smithi046.stdout:mon.smithi073 smithi073 running (4m) 67s ago 4m 40.7M 2048M 18.2.2-1260-g5403a021 267a473d505d 77769d199765 2024-04-27T04:01:08.612 INFO:teuthology.orchestra.run.smithi046.stdout:node-exporter.smithi046 smithi046 *:9100 running (5m) 2m ago 5m 16.4M - 1.5.0 0da6a335fe13 02c01746eff1 2024-04-27T04:01:08.612 INFO:teuthology.orchestra.run.smithi046.stdout:node-exporter.smithi073 smithi073 *:9100 running (4m) 67s ago 4m 14.9M - 1.5.0 0da6a335fe13 377ef4267056 2024-04-27T04:01:08.612 INFO:teuthology.orchestra.run.smithi046.stdout:osd.0 smithi046 running (3m) 2m ago 3m 50.0M 3948M 18.2.2-1260-g5403a021 267a473d505d 563c995c0df6 2024-04-27T04:01:08.613 INFO:teuthology.orchestra.run.smithi046.stdout:osd.1 smithi046 running (2m) 2m ago 2m 40.5M 3948M 18.2.2-1260-g5403a021 267a473d505d 5673c75e53bb 2024-04-27T04:01:08.613 INFO:teuthology.orchestra.run.smithi046.stdout:osd.2 smithi046 running (2m) 2m ago 2m 41.7M 3948M 18.2.2-1260-g5403a021 267a473d505d 9787b5012842 2024-04-27T04:01:08.613 INFO:teuthology.orchestra.run.smithi046.stdout:osd.3 smithi073 running (2m) 67s ago 2m 41.7M 4970M 18.2.2-1260-g5403a021 267a473d505d 0c89b358284d 2024-04-27T04:01:08.613 INFO:teuthology.orchestra.run.smithi046.stdout:osd.4 smithi073 running (99s) 67s ago 99s 48.3M 4970M 18.2.2-1260-g5403a021 267a473d505d 26f06c87c31e 2024-04-27T04:01:08.613 INFO:teuthology.orchestra.run.smithi046.stdout:osd.5 smithi073 running (73s) 67s ago 72s 10.8M 4970M 18.2.2-1260-g5403a021 267a473d505d e1ce06e0da9b 2024-04-27T04:01:08.613 INFO:teuthology.orchestra.run.smithi046.stdout:prometheus.smithi046 smithi046 *:9095 running (3m) 2m ago 4m 39.9M - 2.43.0 a07b618ecd1d e1a7d8656893 2024-04-27T04:01:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.613+0000 7f44c141c700 1 -- 172.21.15.46:0/4179315027 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f44a406cc90 msgr2=0x7f44a406f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.613+0000 7f44c141c700 1 --2- 172.21.15.46:0/4179315027 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f44a406cc90 0x7f44a406f150 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f44b00051c0 tx=0x7f44b000b540 comp rx=0 tx=0).stop 2024-04-27T04:01:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.613+0000 7f44c141c700 1 -- 172.21.15.46:0/4179315027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44bc100c30 msgr2=0x7f44bc196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.613+0000 7f44c141c700 1 --2- 172.21.15.46:0/4179315027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44bc100c30 0x7f44bc196640 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f44ac00c910 tx=0x7f44ac00ccd0 comp rx=0 tx=0).stop 2024-04-27T04:01:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.613+0000 7f44c141c700 1 -- 172.21.15.46:0/4179315027 shutdown_connections 2024-04-27T04:01:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.613+0000 7f44c141c700 1 --2- 172.21.15.46:0/4179315027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44bc100c30 0x7f44bc196640 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.613+0000 7f44c141c700 1 --2- 172.21.15.46:0/4179315027 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f44a406cc90 0x7f44a406f150 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.613+0000 7f44c141c700 1 --2- 172.21.15.46:0/4179315027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44bc101e30 0x7f44bc196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.613+0000 7f44c141c700 1 -- 172.21.15.46:0/4179315027 >> 172.21.15.46:0/4179315027 conn(0x7f44bc0fc1d0 msgr2=0x7f44bc1053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.613+0000 7f44c141c700 1 -- 172.21.15.46:0/4179315027 shutdown_connections 2024-04-27T04:01:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:08.613+0000 7f44c141c700 1 -- 172.21.15.46:0/4179315027 wait complete. 2024-04-27T04:01:08.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:08 smithi046 ceph-mon[26899]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:09.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:08 smithi073 ceph-mon[34160]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:09.218 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-27T04:01:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:01:09.802 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:09 smithi046 ceph-mon[26899]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:01:10.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:09 smithi073 ceph-mon[34160]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:01:10.879 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:10 smithi046 ceph-mon[26899]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:11.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:10 smithi073 ceph-mon[34160]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.102+0000 7f262f94b700 1 -- 172.21.15.46:0/4190261993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2628101e30 msgr2=0x7f2628102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.102+0000 7f262f94b700 1 --2- 172.21.15.46:0/4190261993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2628101e30 0x7f2628102290 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2618009910 tx=0x7f2618009c20 comp rx=0 tx=0).stop 2024-04-27T04:01:11.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.102+0000 7f262f94b700 1 -- 172.21.15.46:0/4190261993 shutdown_connections 2024-04-27T04:01:11.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.102+0000 7f262f94b700 1 --2- 172.21.15.46:0/4190261993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2628101e30 0x7f2628102290 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:11.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.102+0000 7f262f94b700 1 --2- 172.21.15.46:0/4190261993 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2628100c30 0x7f2628101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:11.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.102+0000 7f262f94b700 1 -- 172.21.15.46:0/4190261993 >> 172.21.15.46:0/4190261993 conn(0x7f26280fc1d0 msgr2=0x7f26280fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:11.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.102+0000 7f262f94b700 1 -- 172.21.15.46:0/4190261993 shutdown_connections 2024-04-27T04:01:11.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.102+0000 7f262f94b700 1 -- 172.21.15.46:0/4190261993 wait complete. 2024-04-27T04:01:11.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.103+0000 7f262f94b700 1 Processor -- start 2024-04-27T04:01:11.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.103+0000 7f262f94b700 1 -- start start 2024-04-27T04:01:11.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.103+0000 7f262f94b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2628100c30 0x7f2628196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:11.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.103+0000 7f262f94b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2628101e30 0x7f2628196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:11.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.103+0000 7f262f94b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26281971c0 con 0x7f2628100c30 2024-04-27T04:01:11.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.103+0000 7f262f94b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2628197300 con 0x7f2628101e30 2024-04-27T04:01:11.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.103+0000 7f262d6e7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2628100c30 0x7f2628196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:11.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.103+0000 7f262d6e7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2628100c30 0x7f2628196660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47160/0 (socket says 172.21.15.46:47160) 2024-04-27T04:01:11.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.103+0000 7f262d6e7700 1 -- 172.21.15.46:0/2127530355 learned_addr learned my addr 172.21.15.46:0/2127530355 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:11.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.103+0000 7f262cee6700 1 --2- 172.21.15.46:0/2127530355 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2628101e30 0x7f2628196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:11.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.103+0000 7f262d6e7700 1 -- 172.21.15.46:0/2127530355 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2628101e30 msgr2=0x7f2628196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:11.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.103+0000 7f262d6e7700 1 --2- 172.21.15.46:0/2127530355 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2628101e30 0x7f2628196ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:11.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.103+0000 7f262d6e7700 1 -- 172.21.15.46:0/2127530355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26180095f0 con 0x7f2628100c30 2024-04-27T04:01:11.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.103+0000 7f262cee6700 1 --2- 172.21.15.46:0/2127530355 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2628101e30 0x7f2628196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T04:01:11.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.103+0000 7f262d6e7700 1 --2- 172.21.15.46:0/2127530355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2628100c30 0x7f2628196660 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f262400d900 tx=0x7f262400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:11.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.104+0000 7f261e7fc700 1 -- 172.21.15.46:0/2127530355 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2624009900 con 0x7f2628100c30 2024-04-27T04:01:11.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.104+0000 7f261e7fc700 1 -- 172.21.15.46:0/2127530355 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2624010460 con 0x7f2628100c30 2024-04-27T04:01:11.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.104+0000 7f262f94b700 1 -- 172.21.15.46:0/2127530355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262819bd10 con 0x7f2628100c30 2024-04-27T04:01:11.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.104+0000 7f261e7fc700 1 -- 172.21.15.46:0/2127530355 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f262400f5b0 con 0x7f2628100c30 2024-04-27T04:01:11.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.104+0000 7f262f94b700 1 -- 172.21.15.46:0/2127530355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f262819c280 con 0x7f2628100c30 2024-04-27T04:01:11.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.105+0000 7f262f94b700 1 -- 172.21.15.46:0/2127530355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262804ec20 con 0x7f2628100c30 2024-04-27T04:01:11.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.105+0000 7f261e7fc700 1 -- 172.21.15.46:0/2127530355 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f26240105d0 con 0x7f2628100c30 2024-04-27T04:01:11.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.106+0000 7f261e7fc700 1 --2- 172.21.15.46:0/2127530355 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f261406cce0 0x7f261406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:11.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.106+0000 7f261e7fc700 1 -- 172.21.15.46:0/2127530355 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f262408bd60 con 0x7f2628100c30 2024-04-27T04:01:11.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.106+0000 7f262cee6700 1 --2- 172.21.15.46:0/2127530355 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f261406cce0 0x7f261406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:11.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.106+0000 7f262cee6700 1 --2- 172.21.15.46:0/2127530355 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f261406cce0 0x7f261406f1a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2618005b40 tx=0x7f2618005ab0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:11.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.110+0000 7f261e7fc700 1 -- 172.21.15.46:0/2127530355 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f262804ec20 con 0x7f2628100c30 2024-04-27T04:01:11.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.321+0000 7f262f94b700 1 -- 172.21.15.46:0/2127530355 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f262804ec20 con 0x7f261406cce0 2024-04-27T04:01:11.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.325+0000 7f261e7fc700 1 -- 172.21.15.46:0/2127530355 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1290 (secure 0 0 0) 0x7f262804ec20 con 0x7f261406cce0 2024-04-27T04:01:11.326 INFO:teuthology.orchestra.run.smithi046.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-27T04:01:11.326 INFO:teuthology.orchestra.run.smithi046.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-27T04:01:11.326 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-exporter 2/2 2m ago 6m * 2024-04-27T04:01:11.326 INFO:teuthology.orchestra.run.smithi046.stdout:crash 2/2 2m ago 6m * 2024-04-27T04:01:11.327 INFO:teuthology.orchestra.run.smithi046.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-27T04:01:11.327 INFO:teuthology.orchestra.run.smithi046.stdout:mgr 2/2 2m ago 6m count:2 2024-04-27T04:01:11.327 INFO:teuthology.orchestra.run.smithi046.stdout:mon 2/2 2m ago 5m smithi046:172.21.15.46=smithi046;smithi073:172.21.15.73=smithi073;count:2 2024-04-27T04:01:11.327 INFO:teuthology.orchestra.run.smithi046.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-27T04:01:11.327 INFO:teuthology.orchestra.run.smithi046.stdout:osd 6 2m ago - 2024-04-27T04:01:11.327 INFO:teuthology.orchestra.run.smithi046.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-27T04:01:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.328+0000 7f262f94b700 1 -- 172.21.15.46:0/2127530355 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f261406cce0 msgr2=0x7f261406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.328+0000 7f262f94b700 1 --2- 172.21.15.46:0/2127530355 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f261406cce0 0x7f261406f1a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2618005b40 tx=0x7f2618005ab0 comp rx=0 tx=0).stop 2024-04-27T04:01:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.328+0000 7f262f94b700 1 -- 172.21.15.46:0/2127530355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2628100c30 msgr2=0x7f2628196660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.328+0000 7f262f94b700 1 --2- 172.21.15.46:0/2127530355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2628100c30 0x7f2628196660 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f262400d900 tx=0x7f262400dcc0 comp rx=0 tx=0).stop 2024-04-27T04:01:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.328+0000 7f262f94b700 1 -- 172.21.15.46:0/2127530355 shutdown_connections 2024-04-27T04:01:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.328+0000 7f262f94b700 1 --2- 172.21.15.46:0/2127530355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2628100c30 0x7f2628196660 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.328+0000 7f262f94b700 1 --2- 172.21.15.46:0/2127530355 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f261406cce0 0x7f261406f1a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.328+0000 7f262f94b700 1 --2- 172.21.15.46:0/2127530355 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2628101e30 0x7f2628196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.328+0000 7f262f94b700 1 -- 172.21.15.46:0/2127530355 >> 172.21.15.46:0/2127530355 conn(0x7f26280fc1d0 msgr2=0x7f26281053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.328+0000 7f262f94b700 1 -- 172.21.15.46:0/2127530355 shutdown_connections 2024-04-27T04:01:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:11.328+0000 7f262f94b700 1 -- 172.21.15.46:0/2127530355 wait complete. 2024-04-27T04:01:11.999 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-27T04:01:12.212 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:01:12.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:12 smithi046 ceph-mon[26899]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:12.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:12 smithi046 ceph-mon[26899]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:01:13.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:12 smithi073 ceph-mon[34160]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:13.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:12 smithi073 ceph-mon[34160]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:01:13.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.870+0000 7f36da4f8700 1 -- 172.21.15.46:0/1799763565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d4100c30 msgr2=0x7f36d4101050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:13.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.870+0000 7f36da4f8700 1 --2- 172.21.15.46:0/1799763565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d4100c30 0x7f36d4101050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f36c4009910 tx=0x7f36c4009c20 comp rx=0 tx=0).stop 2024-04-27T04:01:13.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.870+0000 7f36da4f8700 1 -- 172.21.15.46:0/1799763565 shutdown_connections 2024-04-27T04:01:13.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.870+0000 7f36da4f8700 1 --2- 172.21.15.46:0/1799763565 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36d4101e30 0x7f36d4102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:13.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.870+0000 7f36da4f8700 1 --2- 172.21.15.46:0/1799763565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d4100c30 0x7f36d4101050 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:13.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.870+0000 7f36da4f8700 1 -- 172.21.15.46:0/1799763565 >> 172.21.15.46:0/1799763565 conn(0x7f36d40fc1d0 msgr2=0x7f36d40fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:13.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.870+0000 7f36da4f8700 1 -- 172.21.15.46:0/1799763565 shutdown_connections 2024-04-27T04:01:13.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.870+0000 7f36da4f8700 1 -- 172.21.15.46:0/1799763565 wait complete. 2024-04-27T04:01:13.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.871+0000 7f36da4f8700 1 Processor -- start 2024-04-27T04:01:13.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.871+0000 7f36da4f8700 1 -- start start 2024-04-27T04:01:13.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.871+0000 7f36da4f8700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36d4100c30 0x7f36d4198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:13.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.871+0000 7f36da4f8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d4101e30 0x7f36d4198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:13.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.871+0000 7f36da4f8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d41993a0 con 0x7f36d4101e30 2024-04-27T04:01:13.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.871+0000 7f36da4f8700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d41994e0 con 0x7f36d4100c30 2024-04-27T04:01:13.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.872+0000 7f36d37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d4101e30 0x7f36d4198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:13.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.872+0000 7f36d37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d4101e30 0x7f36d4198d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47174/0 (socket says 172.21.15.46:47174) 2024-04-27T04:01:13.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.872+0000 7f36d37fe700 1 -- 172.21.15.46:0/1253781361 learned_addr learned my addr 172.21.15.46:0/1253781361 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:13.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.872+0000 7f36d3fff700 1 --2- 172.21.15.46:0/1253781361 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36d4100c30 0x7f36d4198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:13.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.872+0000 7f36d37fe700 1 -- 172.21.15.46:0/1253781361 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36d4100c30 msgr2=0x7f36d4198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:13.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.872+0000 7f36d37fe700 1 --2- 172.21.15.46:0/1253781361 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36d4100c30 0x7f36d4198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:13.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.872+0000 7f36d37fe700 1 -- 172.21.15.46:0/1253781361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c40095f0 con 0x7f36d4101e30 2024-04-27T04:01:13.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.872+0000 7f36d3fff700 1 --2- 172.21.15.46:0/1253781361 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36d4100c30 0x7f36d4198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T04:01:13.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.872+0000 7f36d37fe700 1 --2- 172.21.15.46:0/1253781361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d4101e30 0x7f36d4198d80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f36c800c8a0 tx=0x7f36c800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:13.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.872+0000 7f36d17fa700 1 -- 172.21.15.46:0/1253781361 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36c800cea0 con 0x7f36d4101e30 2024-04-27T04:01:13.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.872+0000 7f36d17fa700 1 -- 172.21.15.46:0/1253781361 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36c8004830 con 0x7f36d4101e30 2024-04-27T04:01:13.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.872+0000 7f36da4f8700 1 -- 172.21.15.46:0/1253781361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d419df40 con 0x7f36d4101e30 2024-04-27T04:01:13.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.872+0000 7f36d17fa700 1 -- 172.21.15.46:0/1253781361 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36c80055b0 con 0x7f36d4101e30 2024-04-27T04:01:13.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.872+0000 7f36da4f8700 1 -- 172.21.15.46:0/1253781361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d419e4e0 con 0x7f36d4101e30 2024-04-27T04:01:13.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.873+0000 7f36da4f8700 1 -- 172.21.15.46:0/1253781361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d4106800 con 0x7f36d4101e30 2024-04-27T04:01:13.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.874+0000 7f36d17fa700 1 -- 172.21.15.46:0/1253781361 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f36c80049a0 con 0x7f36d4101e30 2024-04-27T04:01:13.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.874+0000 7f36d17fa700 1 --2- 172.21.15.46:0/1253781361 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f36bc06ccf0 0x7f36bc06f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:13.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.875+0000 7f36d3fff700 1 --2- 172.21.15.46:0/1253781361 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f36bc06ccf0 0x7f36bc06f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:13.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.875+0000 7f36d17fa700 1 -- 172.21.15.46:0/1253781361 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f36c808bc10 con 0x7f36d4101e30 2024-04-27T04:01:13.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.875+0000 7f36d3fff700 1 --2- 172.21.15.46:0/1253781361 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f36bc06ccf0 0x7f36bc06f1b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f36c4006010 tx=0x7f36c400b560 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:13.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:13.879+0000 7f36d17fa700 1 -- 172.21.15.46:0/1253781361 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f36c8059d50 con 0x7f36d4101e30 2024-04-27T04:01:14.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:14.088+0000 7f36da4f8700 1 -- 172.21.15.46:0/1253781361 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f36d402cdb0 con 0x7f36bc06ccf0 2024-04-27T04:01:14.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:14.090+0000 7f36d17fa700 1 -- 172.21.15.46:0/1253781361 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+145 (secure 0 0 0) 0x7f36d402cdb0 con 0x7f36bc06ccf0 2024-04-27T04:01:14.091 INFO:teuthology.orchestra.run.smithi046.stdout:HOST ADDR LABELS STATUS 2024-04-27T04:01:14.091 INFO:teuthology.orchestra.run.smithi046.stdout:smithi046 172.21.15.46 2024-04-27T04:01:14.091 INFO:teuthology.orchestra.run.smithi046.stdout:smithi073 172.21.15.73 2024-04-27T04:01:14.091 INFO:teuthology.orchestra.run.smithi046.stdout:2 hosts in cluster 2024-04-27T04:01:14.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:14.092+0000 7f36da4f8700 1 -- 172.21.15.46:0/1253781361 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f36bc06ccf0 msgr2=0x7f36bc06f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:14.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:14.092+0000 7f36da4f8700 1 --2- 172.21.15.46:0/1253781361 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f36bc06ccf0 0x7f36bc06f1b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f36c4006010 tx=0x7f36c400b560 comp rx=0 tx=0).stop 2024-04-27T04:01:14.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:14.092+0000 7f36da4f8700 1 -- 172.21.15.46:0/1253781361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d4101e30 msgr2=0x7f36d4198d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:14.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:14.092+0000 7f36da4f8700 1 --2- 172.21.15.46:0/1253781361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d4101e30 0x7f36d4198d80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f36c800c8a0 tx=0x7f36c800cc60 comp rx=0 tx=0).stop 2024-04-27T04:01:14.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:14.092+0000 7f36da4f8700 1 -- 172.21.15.46:0/1253781361 shutdown_connections 2024-04-27T04:01:14.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:14.092+0000 7f36da4f8700 1 --2- 172.21.15.46:0/1253781361 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f36bc06ccf0 0x7f36bc06f1b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:14.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:14.092+0000 7f36da4f8700 1 --2- 172.21.15.46:0/1253781361 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36d4100c30 0x7f36d4198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:14.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:14.092+0000 7f36da4f8700 1 --2- 172.21.15.46:0/1253781361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d4101e30 0x7f36d4198d80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:14.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:14.092+0000 7f36da4f8700 1 -- 172.21.15.46:0/1253781361 >> 172.21.15.46:0/1253781361 conn(0x7f36d40fc1d0 msgr2=0x7f36d4073490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:14.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:14.092+0000 7f36da4f8700 1 -- 172.21.15.46:0/1253781361 shutdown_connections 2024-04-27T04:01:14.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:14.092+0000 7f36da4f8700 1 -- 172.21.15.46:0/1253781361 wait complete. 2024-04-27T04:01:14.785 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-27T04:01:14.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:14 smithi046 ceph-mon[26899]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:14.994 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:01:15.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:14 smithi073 ceph-mon[34160]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:15.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:15 smithi046 ceph-mon[26899]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:01:16.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:15 smithi073 ceph-mon[34160]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:01:16.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.769+0000 7f3198c07700 1 -- 172.21.15.46:0/3778246957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3194101e30 msgr2=0x7f3194102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:16.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.769+0000 7f3198c07700 1 --2- 172.21.15.46:0/3778246957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3194101e30 0x7f3194102290 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3188009910 tx=0x7f3188009c20 comp rx=0 tx=0).stop 2024-04-27T04:01:16.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.769+0000 7f3198c07700 1 -- 172.21.15.46:0/3778246957 shutdown_connections 2024-04-27T04:01:16.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.769+0000 7f3198c07700 1 --2- 172.21.15.46:0/3778246957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3194101e30 0x7f3194102290 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:16.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.769+0000 7f3198c07700 1 --2- 172.21.15.46:0/3778246957 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3194100c30 0x7f3194101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:16.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.769+0000 7f3198c07700 1 -- 172.21.15.46:0/3778246957 >> 172.21.15.46:0/3778246957 conn(0x7f31940fc1d0 msgr2=0x7f31940fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:16.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.769+0000 7f3198c07700 1 -- 172.21.15.46:0/3778246957 shutdown_connections 2024-04-27T04:01:16.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.769+0000 7f3198c07700 1 -- 172.21.15.46:0/3778246957 wait complete. 2024-04-27T04:01:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.770+0000 7f3198c07700 1 Processor -- start 2024-04-27T04:01:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.770+0000 7f3198c07700 1 -- start start 2024-04-27T04:01:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.770+0000 7f3198c07700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3194100c30 0x7f3194198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.770+0000 7f3198c07700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3194101e30 0x7f3194198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.770+0000 7f3198c07700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31941993a0 con 0x7f3194101e30 2024-04-27T04:01:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.770+0000 7f3198c07700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31941994e0 con 0x7f3194100c30 2024-04-27T04:01:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.770+0000 7f3191d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3194101e30 0x7f3194198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.770+0000 7f3191d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3194101e30 0x7f3194198d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47186/0 (socket says 172.21.15.46:47186) 2024-04-27T04:01:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.770+0000 7f3191d9b700 1 -- 172.21.15.46:0/2071742573 learned_addr learned my addr 172.21.15.46:0/2071742573 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.770+0000 7f319259c700 1 --2- 172.21.15.46:0/2071742573 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3194100c30 0x7f3194198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.770+0000 7f3191d9b700 1 -- 172.21.15.46:0/2071742573 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3194100c30 msgr2=0x7f3194198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.770+0000 7f3191d9b700 1 --2- 172.21.15.46:0/2071742573 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3194100c30 0x7f3194198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.770+0000 7f3191d9b700 1 -- 172.21.15.46:0/2071742573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31880095f0 con 0x7f3194101e30 2024-04-27T04:01:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.770+0000 7f3191d9b700 1 --2- 172.21.15.46:0/2071742573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3194101e30 0x7f3194198d80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f31880092e0 tx=0x7f318800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.770+0000 7f31837fe700 1 -- 172.21.15.46:0/2071742573 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f318801d070 con 0x7f3194101e30 2024-04-27T04:01:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.771+0000 7f31837fe700 1 -- 172.21.15.46:0/2071742573 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f318800f460 con 0x7f3194101e30 2024-04-27T04:01:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.771+0000 7f3198c07700 1 -- 172.21.15.46:0/2071742573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f319419dee0 con 0x7f3194101e30 2024-04-27T04:01:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.771+0000 7f31837fe700 1 -- 172.21.15.46:0/2071742573 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3188021640 con 0x7f3194101e30 2024-04-27T04:01:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.771+0000 7f3198c07700 1 -- 172.21.15.46:0/2071742573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319419e420 con 0x7f3194101e30 2024-04-27T04:01:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.772+0000 7f3198c07700 1 -- 172.21.15.46:0/2071742573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3194106800 con 0x7f3194101e30 2024-04-27T04:01:16.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.772+0000 7f31837fe700 1 -- 172.21.15.46:0/2071742573 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f318800f5d0 con 0x7f3194101e30 2024-04-27T04:01:16.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.773+0000 7f31837fe700 1 --2- 172.21.15.46:0/2071742573 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f317c06cce0 0x7f317c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:16.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.773+0000 7f319259c700 1 --2- 172.21.15.46:0/2071742573 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f317c06cce0 0x7f317c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:16.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.773+0000 7f31837fe700 1 -- 172.21.15.46:0/2071742573 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f318808d1f0 con 0x7f3194101e30 2024-04-27T04:01:16.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.773+0000 7f319259c700 1 --2- 172.21.15.46:0/2071742573 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f317c06cce0 0x7f317c06f1a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3194101c90 tx=0x7f318400a380 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:16.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.777+0000 7f31837fe700 1 -- 172.21.15.46:0/2071742573 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f318805b390 con 0x7f3194101e30 2024-04-27T04:01:16.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:16 smithi046 ceph-mon[26899]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:16.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:16 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:01:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.988+0000 7f3198c07700 1 -- 172.21.15.46:0/2071742573 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f319402cea0 con 0x7f317c06cce0 2024-04-27T04:01:16.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.990+0000 7f31837fe700 1 -- 172.21.15.46:0/2071742573 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f319402cea0 con 0x7f317c06cce0 2024-04-27T04:01:16.991 INFO:teuthology.orchestra.run.smithi046.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-27T04:01:16.991 INFO:teuthology.orchestra.run.smithi046.stdout:smithi046 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7512009A480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-04-27T04:01:16.991 INFO:teuthology.orchestra.run.smithi046.stdout:smithi073 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040115400BGN 372G No 72s ago Has a FileSystem, LVM detected 2024-04-27T04:01:16.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.992+0000 7f3198c07700 1 -- 172.21.15.46:0/2071742573 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f317c06cce0 msgr2=0x7f317c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.992+0000 7f3198c07700 1 --2- 172.21.15.46:0/2071742573 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f317c06cce0 0x7f317c06f1a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3194101c90 tx=0x7f318400a380 comp rx=0 tx=0).stop 2024-04-27T04:01:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.992+0000 7f3198c07700 1 -- 172.21.15.46:0/2071742573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3194101e30 msgr2=0x7f3194198d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.992+0000 7f3198c07700 1 --2- 172.21.15.46:0/2071742573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3194101e30 0x7f3194198d80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f31880092e0 tx=0x7f318800bd70 comp rx=0 tx=0).stop 2024-04-27T04:01:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.992+0000 7f3198c07700 1 -- 172.21.15.46:0/2071742573 shutdown_connections 2024-04-27T04:01:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.992+0000 7f3198c07700 1 --2- 172.21.15.46:0/2071742573 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f317c06cce0 0x7f317c06f1a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.992+0000 7f3198c07700 1 --2- 172.21.15.46:0/2071742573 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3194100c30 0x7f3194198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.992+0000 7f3198c07700 1 --2- 172.21.15.46:0/2071742573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3194101e30 0x7f3194198d80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:16.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.992+0000 7f3198c07700 1 -- 172.21.15.46:0/2071742573 >> 172.21.15.46:0/2071742573 conn(0x7f31940fc1d0 msgr2=0x7f3194073490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:16.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.992+0000 7f3198c07700 1 -- 172.21.15.46:0/2071742573 shutdown_connections 2024-04-27T04:01:16.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:16.992+0000 7f3198c07700 1 -- 172.21.15.46:0/2071742573 wait complete. 2024-04-27T04:01:17.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:16 smithi073 ceph-mon[34160]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:17.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:16 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:01:17.668 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T04:01:17.678 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi046.front.sepia.ceph.com 2024-04-27T04:01:17.678 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-27T04:01:17.897 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:01:18.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:18 smithi046 ceph-mon[26899]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:18.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:18 smithi046 ceph-mon[26899]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:01:19.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:18 smithi073 ceph-mon[34160]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:19.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:18 smithi073 ceph-mon[34160]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:01:19.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.594+0000 7f96f68b8700 1 -- 172.21.15.46:0/2201874385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96f0103fd0 msgr2=0x7f96f01063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:19.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.594+0000 7f96f68b8700 1 --2- 172.21.15.46:0/2201874385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96f0103fd0 0x7f96f01063c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009910 tx=0x7f96e4009c20 comp rx=0 tx=0).stop 2024-04-27T04:01:19.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.594+0000 7f96f68b8700 1 -- 172.21.15.46:0/2201874385 shutdown_connections 2024-04-27T04:01:19.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.594+0000 7f96f68b8700 1 --2- 172.21.15.46:0/2201874385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96f0103fd0 0x7f96f01063c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:19.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.594+0000 7f96f68b8700 1 --2- 172.21.15.46:0/2201874385 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96f01016a0 0x7f96f0103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:19.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.594+0000 7f96f68b8700 1 -- 172.21.15.46:0/2201874385 >> 172.21.15.46:0/2201874385 conn(0x7f96f00fb0b0 msgr2=0x7f96f00fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:19.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.594+0000 7f96f68b8700 1 -- 172.21.15.46:0/2201874385 shutdown_connections 2024-04-27T04:01:19.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.594+0000 7f96f68b8700 1 -- 172.21.15.46:0/2201874385 wait complete. 2024-04-27T04:01:19.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.595+0000 7f96f68b8700 1 Processor -- start 2024-04-27T04:01:19.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.595+0000 7f96f68b8700 1 -- start start 2024-04-27T04:01:19.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.595+0000 7f96f68b8700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96f01016a0 0x7f96f01965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:19.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.595+0000 7f96f68b8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96f0103fd0 0x7f96f0196b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:19.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.595+0000 7f96f68b8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f01970a0 con 0x7f96f0103fd0 2024-04-27T04:01:19.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.595+0000 7f96f68b8700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f01971e0 con 0x7f96f01016a0 2024-04-27T04:01:19.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.595+0000 7f96ef7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96f0103fd0 0x7f96f0196b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:19.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.595+0000 7f96ef7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96f0103fd0 0x7f96f0196b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47214/0 (socket says 172.21.15.46:47214) 2024-04-27T04:01:19.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.595+0000 7f96ef7fe700 1 -- 172.21.15.46:0/3912024547 learned_addr learned my addr 172.21.15.46:0/3912024547 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:19.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.596+0000 7f96ef7fe700 1 -- 172.21.15.46:0/3912024547 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96f01016a0 msgr2=0x7f96f01965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:19.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.596+0000 7f96effff700 1 --2- 172.21.15.46:0/3912024547 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96f01016a0 0x7f96f01965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:19.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.596+0000 7f96ef7fe700 1 --2- 172.21.15.46:0/3912024547 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96f01016a0 0x7f96f01965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:19.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.596+0000 7f96ef7fe700 1 -- 172.21.15.46:0/3912024547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e40095f0 con 0x7f96f0103fd0 2024-04-27T04:01:19.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.596+0000 7f96effff700 1 --2- 172.21.15.46:0/3912024547 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96f01016a0 0x7f96f01965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T04:01:19.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.596+0000 7f96ef7fe700 1 --2- 172.21.15.46:0/3912024547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96f0103fd0 0x7f96f0196b10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f96e4006010 tx=0x7f96e400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:19.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.596+0000 7f96ed7fa700 1 -- 172.21.15.46:0/3912024547 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96e401d070 con 0x7f96f0103fd0 2024-04-27T04:01:19.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.596+0000 7f96ed7fa700 1 -- 172.21.15.46:0/3912024547 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96e400f460 con 0x7f96f0103fd0 2024-04-27T04:01:19.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.596+0000 7f96ed7fa700 1 -- 172.21.15.46:0/3912024547 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96e4021640 con 0x7f96f0103fd0 2024-04-27T04:01:19.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.596+0000 7f96f68b8700 1 -- 172.21.15.46:0/3912024547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f019bbf0 con 0x7f96f0103fd0 2024-04-27T04:01:19.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.596+0000 7f96f68b8700 1 -- 172.21.15.46:0/3912024547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f019c0e0 con 0x7f96f0103fd0 2024-04-27T04:01:19.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.597+0000 7f96f68b8700 1 -- 172.21.15.46:0/3912024547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f004ebb0 con 0x7f96f0103fd0 2024-04-27T04:01:19.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.598+0000 7f96ed7fa700 1 -- 172.21.15.46:0/3912024547 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f96e400fab0 con 0x7f96f0103fd0 2024-04-27T04:01:19.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.598+0000 7f96ed7fa700 1 --2- 172.21.15.46:0/3912024547 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f96d806cce0 0x7f96d806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:19.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.598+0000 7f96ed7fa700 1 -- 172.21.15.46:0/3912024547 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f96e408d1e0 con 0x7f96f0103fd0 2024-04-27T04:01:19.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.598+0000 7f96effff700 1 --2- 172.21.15.46:0/3912024547 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f96d806cce0 0x7f96d806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:19.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.599+0000 7f96effff700 1 --2- 172.21.15.46:0/3912024547 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f96d806cce0 0x7f96d806f1a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f96e0005fd0 tx=0x7f96e0005e20 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:19.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.603+0000 7f96ed7fa700 1 -- 172.21.15.46:0/3912024547 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f96e405b400 con 0x7f96f0103fd0 2024-04-27T04:01:19.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:19.837+0000 7f96f68b8700 1 -- 172.21.15.46:0/3912024547 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f96f0105eb0 con 0x7f96d806cce0 2024-04-27T04:01:20.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:20 smithi046 ceph-mon[26899]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:20.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:20 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-27T04:01:21.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:20 smithi073 ceph-mon[34160]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:21.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:20 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-27T04:01:21.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:21.704+0000 7f96ed7fa700 1 -- 172.21.15.46:0/3912024547 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f96f0105eb0 con 0x7f96d806cce0 2024-04-27T04:01:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:21.707+0000 7f96f68b8700 1 -- 172.21.15.46:0/3912024547 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f96d806cce0 msgr2=0x7f96d806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:21.707+0000 7f96f68b8700 1 --2- 172.21.15.46:0/3912024547 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f96d806cce0 0x7f96d806f1a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f96e0005fd0 tx=0x7f96e0005e20 comp rx=0 tx=0).stop 2024-04-27T04:01:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:21.707+0000 7f96f68b8700 1 -- 172.21.15.46:0/3912024547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96f0103fd0 msgr2=0x7f96f0196b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:21.707+0000 7f96f68b8700 1 --2- 172.21.15.46:0/3912024547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96f0103fd0 0x7f96f0196b10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f96e4006010 tx=0x7f96e400be20 comp rx=0 tx=0).stop 2024-04-27T04:01:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:21.707+0000 7f96f68b8700 1 -- 172.21.15.46:0/3912024547 shutdown_connections 2024-04-27T04:01:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:21.707+0000 7f96f68b8700 1 --2- 172.21.15.46:0/3912024547 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f96d806cce0 0x7f96d806f1a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:21.707+0000 7f96f68b8700 1 --2- 172.21.15.46:0/3912024547 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96f01016a0 0x7f96f01965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:21.707+0000 7f96f68b8700 1 --2- 172.21.15.46:0/3912024547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96f0103fd0 0x7f96f0196b10 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:21.707+0000 7f96f68b8700 1 -- 172.21.15.46:0/3912024547 >> 172.21.15.46:0/3912024547 conn(0x7f96f00fb0b0 msgr2=0x7f96f00fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:21.707+0000 7f96f68b8700 1 -- 172.21.15.46:0/3912024547 shutdown_connections 2024-04-27T04:01:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:21.707+0000 7f96f68b8700 1 -- 172.21.15.46:0/3912024547 wait complete. 2024-04-27T04:01:21.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:21 smithi046 ceph-mon[26899]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:01:21.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:21 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-27T04:01:21.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:21 smithi046 ceph-mon[26899]: osdmap e41: 6 total, 6 up, 6 in 2024-04-27T04:01:21.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:21 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-27T04:01:21.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:01:21.676+0000 7f50b2f1c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T04:01:22.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:21 smithi073 ceph-mon[34160]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:01:22.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:21 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-27T04:01:22.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:21 smithi073 ceph-mon[34160]: osdmap e41: 6 total, 6 up, 6 in 2024-04-27T04:01:22.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:21 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-27T04:01:22.423 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-27T04:01:22.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: pgmap v174: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:22.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T04:01:22.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-27T04:01:22.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: osdmap e42: 6 total, 6 up, 6 in 2024-04-27T04:01:22.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-27T04:01:22.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T04:01:22.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-27T04:01:22.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: osdmap e43: 6 total, 6 up, 6 in 2024-04-27T04:01:22.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-27T04:01:22.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: fsmap cephfs:0 2024-04-27T04:01:22.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:22.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:01:22.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:01:22.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:01:22.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:22.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi046.qgmaql", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T04:01:22.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi046.qgmaql", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T04:01:22.925 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:22 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:01:23.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: pgmap v174: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:23.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T04:01:23.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-27T04:01:23.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: osdmap e42: 6 total, 6 up, 6 in 2024-04-27T04:01:23.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-27T04:01:23.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T04:01:23.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-27T04:01:23.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: osdmap e43: 6 total, 6 up, 6 in 2024-04-27T04:01:23.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-27T04:01:23.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: fsmap cephfs:0 2024-04-27T04:01:23.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:23.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:01:23.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:01:23.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:01:23.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:23.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi046.qgmaql", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T04:01:23.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi046.qgmaql", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T04:01:23.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:22 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:01:23.682 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:01:23.916 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:23 smithi046 ceph-mon[26899]: Saving service mds.cephfs spec with placement count:4 2024-04-27T04:01:23.916 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:23 smithi046 ceph-mon[26899]: Deploying daemon mds.cephfs.smithi046.qgmaql on smithi046 2024-04-27T04:01:23.916 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:23 smithi046 ceph-mon[26899]: osdmap e44: 6 total, 6 up, 6 in 2024-04-27T04:01:24.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:23 smithi073 ceph-mon[34160]: Saving service mds.cephfs spec with placement count:4 2024-04-27T04:01:24.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:23 smithi073 ceph-mon[34160]: Deploying daemon mds.cephfs.smithi046.qgmaql on smithi046 2024-04-27T04:01:24.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:23 smithi073 ceph-mon[34160]: osdmap e44: 6 total, 6 up, 6 in 2024-04-27T04:01:24.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:24 smithi046 ceph-mon[26899]: pgmap v178: 65 pgs: 9 creating+peering, 55 unknown, 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:24.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:24 smithi046 ceph-mon[26899]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-27T04:01:24.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:24 smithi046 ceph-mon[26899]: osdmap e45: 6 total, 6 up, 6 in 2024-04-27T04:01:24.976 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:24 smithi073 ceph-mon[34160]: pgmap v178: 65 pgs: 9 creating+peering, 55 unknown, 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:24.977 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:24 smithi073 ceph-mon[34160]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-27T04:01:24.977 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:24 smithi073 ceph-mon[34160]: osdmap e45: 6 total, 6 up, 6 in 2024-04-27T04:01:25.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.574+0000 7f330a599700 1 -- 172.21.15.46:0/3934082227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3304100c30 msgr2=0x7f3304101050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:25.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.574+0000 7f330a599700 1 --2- 172.21.15.46:0/3934082227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3304100c30 0x7f3304101050 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009910 tx=0x7f32f4009c20 comp rx=0 tx=0).stop 2024-04-27T04:01:25.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.574+0000 7f330a599700 1 -- 172.21.15.46:0/3934082227 shutdown_connections 2024-04-27T04:01:25.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.574+0000 7f330a599700 1 --2- 172.21.15.46:0/3934082227 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3304101e30 0x7f3304102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:25.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.574+0000 7f330a599700 1 --2- 172.21.15.46:0/3934082227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3304100c30 0x7f3304101050 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:25.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.574+0000 7f330a599700 1 -- 172.21.15.46:0/3934082227 >> 172.21.15.46:0/3934082227 conn(0x7f33040fc1d0 msgr2=0x7f33040fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:25.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.574+0000 7f330a599700 1 -- 172.21.15.46:0/3934082227 shutdown_connections 2024-04-27T04:01:25.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.574+0000 7f330a599700 1 -- 172.21.15.46:0/3934082227 wait complete. 2024-04-27T04:01:25.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.574+0000 7f330a599700 1 Processor -- start 2024-04-27T04:01:25.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.574+0000 7f330a599700 1 -- start start 2024-04-27T04:01:25.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.575+0000 7f330a599700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3304100c30 0x7f3304196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:25.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.575+0000 7f330a599700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3304101e30 0x7f3304196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:25.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.575+0000 7f330a599700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3304197140 con 0x7f3304100c30 2024-04-27T04:01:25.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.575+0000 7f330a599700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3304197280 con 0x7f3304101e30 2024-04-27T04:01:25.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.575+0000 7f3303fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3304100c30 0x7f3304196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:25.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.575+0000 7f3303fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3304100c30 0x7f3304196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38002/0 (socket says 172.21.15.46:38002) 2024-04-27T04:01:25.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.575+0000 7f3303fff700 1 -- 172.21.15.46:0/778313155 learned_addr learned my addr 172.21.15.46:0/778313155 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:25.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.575+0000 7f33037fe700 1 --2- 172.21.15.46:0/778313155 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3304101e30 0x7f3304196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:25.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.575+0000 7f3303fff700 1 -- 172.21.15.46:0/778313155 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3304101e30 msgr2=0x7f3304196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:25.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.575+0000 7f3303fff700 1 --2- 172.21.15.46:0/778313155 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3304101e30 0x7f3304196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:25.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.575+0000 7f3303fff700 1 -- 172.21.15.46:0/778313155 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f40095f0 con 0x7f3304100c30 2024-04-27T04:01:25.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.575+0000 7f3303fff700 1 --2- 172.21.15.46:0/778313155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3304100c30 0x7f3304196670 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f32f40059d0 tx=0x7f32f400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:25.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.576+0000 7f33017fa700 1 -- 172.21.15.46:0/778313155 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32f401d070 con 0x7f3304100c30 2024-04-27T04:01:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.576+0000 7f330a599700 1 -- 172.21.15.46:0/778313155 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f330419bc40 con 0x7f3304100c30 2024-04-27T04:01:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.576+0000 7f33017fa700 1 -- 172.21.15.46:0/778313155 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32f400f460 con 0x7f3304100c30 2024-04-27T04:01:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.576+0000 7f33017fa700 1 -- 172.21.15.46:0/778313155 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32f4021620 con 0x7f3304100c30 2024-04-27T04:01:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.576+0000 7f330a599700 1 -- 172.21.15.46:0/778313155 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f330419c100 con 0x7f3304100c30 2024-04-27T04:01:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.577+0000 7f330a599700 1 -- 172.21.15.46:0/778313155 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33041907b0 con 0x7f3304100c30 2024-04-27T04:01:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.577+0000 7f33017fa700 1 -- 172.21.15.46:0/778313155 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f32f400fa70 con 0x7f3304100c30 2024-04-27T04:01:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.577+0000 7f33017fa700 1 --2- 172.21.15.46:0/778313155 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f32f006cce0 0x7f32f006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.578+0000 7f33017fa700 1 -- 172.21.15.46:0/778313155 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f32f408d5e0 con 0x7f3304100c30 2024-04-27T04:01:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.578+0000 7f33037fe700 1 --2- 172.21.15.46:0/778313155 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f32f006cce0 0x7f32f006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.578+0000 7f33037fe700 1 --2- 172.21.15.46:0/778313155 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f32f006cce0 0x7f32f006f1a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f32ec005ea0 tx=0x7f32ec005e30 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.582+0000 7f33017fa700 1 -- 172.21.15.46:0/778313155 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f32f405b440 con 0x7f3304100c30 2024-04-27T04:01:25.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.835+0000 7f330a599700 1 -- 172.21.15.46:0/778313155 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f330404ebb0 con 0x7f3304100c30 2024-04-27T04:01:25.835 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:25 smithi046 ceph-mon[26899]: pgmap v180: 65 pgs: 9 creating+peering, 30 unknown, 26 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:25.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.835+0000 7f33017fa700 1 -- 172.21.15.46:0/778313155 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1114 (secure 0 0 0) 0x7f32f405afd0 con 0x7f3304100c30 2024-04-27T04:01:25.836 INFO:teuthology.orchestra.run.smithi046.stdout:e2 2024-04-27T04:01:25.836 INFO:teuthology.orchestra.run.smithi046.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T04:01:25.836 INFO:teuthology.orchestra.run.smithi046.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T04:01:25.836 INFO:teuthology.orchestra.run.smithi046.stdout:legacy client fscid: 1 2024-04-27T04:01:25.836 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:01:25.836 INFO:teuthology.orchestra.run.smithi046.stdout:Filesystem 'cephfs' (1) 2024-04-27T04:01:25.836 INFO:teuthology.orchestra.run.smithi046.stdout:fs_name cephfs 2024-04-27T04:01:25.836 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 2 2024-04-27T04:01:25.836 INFO:teuthology.orchestra.run.smithi046.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-27T04:01:25.836 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-04-27T04:01:21.676811+0000 2024-04-27T04:01:25.837 INFO:teuthology.orchestra.run.smithi046.stdout:modified 2024-04-27T04:01:21.676894+0000 2024-04-27T04:01:25.837 INFO:teuthology.orchestra.run.smithi046.stdout:tableserver 0 2024-04-27T04:01:25.837 INFO:teuthology.orchestra.run.smithi046.stdout:root 0 2024-04-27T04:01:25.837 INFO:teuthology.orchestra.run.smithi046.stdout:session_timeout 60 2024-04-27T04:01:25.837 INFO:teuthology.orchestra.run.smithi046.stdout:session_autoclose 300 2024-04-27T04:01:25.837 INFO:teuthology.orchestra.run.smithi046.stdout:max_file_size 1099511627776 2024-04-27T04:01:25.837 INFO:teuthology.orchestra.run.smithi046.stdout:max_xattr_size 65536 2024-04-27T04:01:25.837 INFO:teuthology.orchestra.run.smithi046.stdout:required_client_features {} 2024-04-27T04:01:25.837 INFO:teuthology.orchestra.run.smithi046.stdout:last_failure 0 2024-04-27T04:01:25.837 INFO:teuthology.orchestra.run.smithi046.stdout:last_failure_osd_epoch 0 2024-04-27T04:01:25.837 INFO:teuthology.orchestra.run.smithi046.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T04:01:25.837 INFO:teuthology.orchestra.run.smithi046.stdout:max_mds 1 2024-04-27T04:01:25.837 INFO:teuthology.orchestra.run.smithi046.stdout:in 2024-04-27T04:01:25.837 INFO:teuthology.orchestra.run.smithi046.stdout:up {} 2024-04-27T04:01:25.837 INFO:teuthology.orchestra.run.smithi046.stdout:failed 2024-04-27T04:01:25.838 INFO:teuthology.orchestra.run.smithi046.stdout:damaged 2024-04-27T04:01:25.838 INFO:teuthology.orchestra.run.smithi046.stdout:stopped 2024-04-27T04:01:25.838 INFO:teuthology.orchestra.run.smithi046.stdout:data_pools [3] 2024-04-27T04:01:25.838 INFO:teuthology.orchestra.run.smithi046.stdout:metadata_pool 2 2024-04-27T04:01:25.838 INFO:teuthology.orchestra.run.smithi046.stdout:inline_data disabled 2024-04-27T04:01:25.838 INFO:teuthology.orchestra.run.smithi046.stdout:balancer 2024-04-27T04:01:25.838 INFO:teuthology.orchestra.run.smithi046.stdout:bal_rank_mask -1 2024-04-27T04:01:25.838 INFO:teuthology.orchestra.run.smithi046.stdout:standby_count_wanted 0 2024-04-27T04:01:25.838 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:01:25.838 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:01:25.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.837+0000 7f330a599700 1 -- 172.21.15.46:0/778313155 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f32f006cce0 msgr2=0x7f32f006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:25.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.837+0000 7f330a599700 1 --2- 172.21.15.46:0/778313155 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f32f006cce0 0x7f32f006f1a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f32ec005ea0 tx=0x7f32ec005e30 comp rx=0 tx=0).stop 2024-04-27T04:01:25.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.837+0000 7f330a599700 1 -- 172.21.15.46:0/778313155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3304100c30 msgr2=0x7f3304196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:25.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.837+0000 7f330a599700 1 --2- 172.21.15.46:0/778313155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3304100c30 0x7f3304196670 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f32f40059d0 tx=0x7f32f400be20 comp rx=0 tx=0).stop 2024-04-27T04:01:25.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.837+0000 7f330a599700 1 -- 172.21.15.46:0/778313155 shutdown_connections 2024-04-27T04:01:25.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.837+0000 7f330a599700 1 --2- 172.21.15.46:0/778313155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3304100c30 0x7f3304196670 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:25.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.837+0000 7f330a599700 1 --2- 172.21.15.46:0/778313155 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f32f006cce0 0x7f32f006f1a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:25.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.837+0000 7f330a599700 1 --2- 172.21.15.46:0/778313155 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3304101e30 0x7f3304196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:25.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.837+0000 7f330a599700 1 -- 172.21.15.46:0/778313155 >> 172.21.15.46:0/778313155 conn(0x7f33040fc1d0 msgr2=0x7f3304105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:25.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.837+0000 7f330a599700 1 -- 172.21.15.46:0/778313155 shutdown_connections 2024-04-27T04:01:25.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:25.837+0000 7f330a599700 1 -- 172.21.15.46:0/778313155 wait complete. 2024-04-27T04:01:25.840 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 2 2024-04-27T04:01:26.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:25 smithi073 ceph-mon[34160]: pgmap v180: 65 pgs: 9 creating+peering, 30 unknown, 26 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:26.697 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T04:01:26.708 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi046.front.sepia.ceph.com 2024-04-27T04:01:26.708 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-04-27T04:01:26.894 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:26 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/778313155' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T04:01:26.894 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:26 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:26.894 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:26 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:26.894 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:26 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:26.894 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:26 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi073.zkoypj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T04:01:26.894 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:26 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi073.zkoypj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T04:01:26.894 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:26 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:01:26.894 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:26 smithi073 ceph-mon[34160]: Deploying daemon mds.cephfs.smithi073.zkoypj on smithi073 2024-04-27T04:01:26.920 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:01:26.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:26 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/778313155' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T04:01:26.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:26 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:26.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:26 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:26.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:26 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:26.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:26 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi073.zkoypj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T04:01:26.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:26 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi073.zkoypj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T04:01:26.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:26 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:01:26.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:26 smithi046 ceph-mon[26899]: Deploying daemon mds.cephfs.smithi073.zkoypj on smithi073 2024-04-27T04:01:28.008 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:27 smithi046 ceph-mon[26899]: mds.? [v2:172.21.15.46:6826/2151398550,v1:172.21.15.46:6827/2151398550] up:boot 2024-04-27T04:01:28.008 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:27 smithi046 ceph-mon[26899]: daemon mds.cephfs.smithi046.qgmaql assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-27T04:01:28.009 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:27 smithi046 ceph-mon[26899]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T04:01:28.009 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:27 smithi046 ceph-mon[26899]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-27T04:01:28.009 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:27 smithi046 ceph-mon[26899]: Cluster is now healthy 2024-04-27T04:01:28.009 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:27 smithi046 ceph-mon[26899]: fsmap cephfs:0 1 up:standby 2024-04-27T04:01:28.009 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:27 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi046.qgmaql"}]: dispatch 2024-04-27T04:01:28.009 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:27 smithi046 ceph-mon[26899]: fsmap cephfs:1 {0=cephfs.smithi046.qgmaql=up:creating} 2024-04-27T04:01:28.009 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:27 smithi046 ceph-mon[26899]: daemon mds.cephfs.smithi046.qgmaql is now active in filesystem cephfs as rank 0 2024-04-27T04:01:28.009 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:27 smithi046 ceph-mon[26899]: pgmap v181: 65 pgs: 65 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:28.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:27 smithi073 ceph-mon[34160]: mds.? [v2:172.21.15.46:6826/2151398550,v1:172.21.15.46:6827/2151398550] up:boot 2024-04-27T04:01:28.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:27 smithi073 ceph-mon[34160]: daemon mds.cephfs.smithi046.qgmaql assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-27T04:01:28.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:27 smithi073 ceph-mon[34160]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T04:01:28.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:27 smithi073 ceph-mon[34160]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-27T04:01:28.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:27 smithi073 ceph-mon[34160]: Cluster is now healthy 2024-04-27T04:01:28.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:27 smithi073 ceph-mon[34160]: fsmap cephfs:0 1 up:standby 2024-04-27T04:01:28.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:27 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi046.qgmaql"}]: dispatch 2024-04-27T04:01:28.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:27 smithi073 ceph-mon[34160]: fsmap cephfs:1 {0=cephfs.smithi046.qgmaql=up:creating} 2024-04-27T04:01:28.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:27 smithi073 ceph-mon[34160]: daemon mds.cephfs.smithi046.qgmaql is now active in filesystem cephfs as rank 0 2024-04-27T04:01:28.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:27 smithi073 ceph-mon[34160]: pgmap v181: 65 pgs: 65 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:01:28.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.526+0000 7f31ab640700 1 -- 172.21.15.46:0/3546799602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31a4100c30 msgr2=0x7f31a4101050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:28.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.526+0000 7f31ab640700 1 --2- 172.21.15.46:0/3546799602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31a4100c30 0x7f31a4101050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009910 tx=0x7f31a0009c20 comp rx=0 tx=0).stop 2024-04-27T04:01:28.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.526+0000 7f31ab640700 1 -- 172.21.15.46:0/3546799602 shutdown_connections 2024-04-27T04:01:28.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.526+0000 7f31ab640700 1 --2- 172.21.15.46:0/3546799602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31a4101e30 0x7f31a4102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:28.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.526+0000 7f31ab640700 1 --2- 172.21.15.46:0/3546799602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31a4100c30 0x7f31a4101050 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:28.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.526+0000 7f31ab640700 1 -- 172.21.15.46:0/3546799602 >> 172.21.15.46:0/3546799602 conn(0x7f31a40fc1d0 msgr2=0x7f31a40fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:28.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.526+0000 7f31ab640700 1 -- 172.21.15.46:0/3546799602 shutdown_connections 2024-04-27T04:01:28.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.526+0000 7f31ab640700 1 -- 172.21.15.46:0/3546799602 wait complete. 2024-04-27T04:01:28.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.526+0000 7f31ab640700 1 Processor -- start 2024-04-27T04:01:28.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.526+0000 7f31ab640700 1 -- start start 2024-04-27T04:01:28.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.527+0000 7f31ab640700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31a4100c30 0x7f31a4196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:28.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.527+0000 7f31ab640700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31a4101e30 0x7f31a4196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:28.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.527+0000 7f31ab640700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a4197180 con 0x7f31a4101e30 2024-04-27T04:01:28.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.527+0000 7f31ab640700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a41972c0 con 0x7f31a4100c30 2024-04-27T04:01:28.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.527+0000 7f31a8bdb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31a4101e30 0x7f31a4196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:28.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.527+0000 7f31a8bdb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31a4101e30 0x7f31a4196b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38026/0 (socket says 172.21.15.46:38026) 2024-04-27T04:01:28.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.527+0000 7f31a8bdb700 1 -- 172.21.15.46:0/470115213 learned_addr learned my addr 172.21.15.46:0/470115213 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:28.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.527+0000 7f31a93dc700 1 --2- 172.21.15.46:0/470115213 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31a4100c30 0x7f31a4196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:28.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.527+0000 7f31a8bdb700 1 -- 172.21.15.46:0/470115213 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31a4100c30 msgr2=0x7f31a4196620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:28.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.527+0000 7f31a8bdb700 1 --2- 172.21.15.46:0/470115213 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31a4100c30 0x7f31a4196620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:28.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.527+0000 7f31a8bdb700 1 -- 172.21.15.46:0/470115213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a00095f0 con 0x7f31a4101e30 2024-04-27T04:01:28.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.527+0000 7f31a8bdb700 1 --2- 172.21.15.46:0/470115213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31a4101e30 0x7f31a4196b60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f319400e920 tx=0x7f319400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:28.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.527+0000 7f319a7fc700 1 -- 172.21.15.46:0/470115213 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3194009790 con 0x7f31a4101e30 2024-04-27T04:01:28.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.527+0000 7f319a7fc700 1 -- 172.21.15.46:0/470115213 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f319400cba0 con 0x7f31a4101e30 2024-04-27T04:01:28.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.527+0000 7f31ab640700 1 -- 172.21.15.46:0/470115213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a419bd20 con 0x7f31a4101e30 2024-04-27T04:01:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.527+0000 7f319a7fc700 1 -- 172.21.15.46:0/470115213 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3194018650 con 0x7f31a4101e30 2024-04-27T04:01:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.528+0000 7f31ab640700 1 -- 172.21.15.46:0/470115213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a419c2f0 con 0x7f31a4101e30 2024-04-27T04:01:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.528+0000 7f31ab640700 1 -- 172.21.15.46:0/470115213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a41907b0 con 0x7f31a4101e30 2024-04-27T04:01:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.529+0000 7f319a7fc700 1 -- 172.21.15.46:0/470115213 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f319400cd10 con 0x7f31a4101e30 2024-04-27T04:01:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.529+0000 7f319a7fc700 1 --2- 172.21.15.46:0/470115213 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f319006cce0 0x7f319006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.530+0000 7f319a7fc700 1 -- 172.21.15.46:0/470115213 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f3194014070 con 0x7f31a4101e30 2024-04-27T04:01:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.530+0000 7f31a93dc700 1 --2- 172.21.15.46:0/470115213 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f319006cce0 0x7f319006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.530+0000 7f31a93dc700 1 --2- 172.21.15.46:0/470115213 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f319006cce0 0x7f319006f1a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f31a000b800 tx=0x7f31a0009f90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.534+0000 7f319a7fc700 1 -- 172.21.15.46:0/470115213 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f3194052c50 con 0x7f31a4101e30 2024-04-27T04:01:28.756 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:28 smithi073 ceph-mon[34160]: mds.? [v2:172.21.15.46:6826/2151398550,v1:172.21.15.46:6827/2151398550] up:active 2024-04-27T04:01:28.756 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:28 smithi073 ceph-mon[34160]: fsmap cephfs:1 {0=cephfs.smithi046.qgmaql=up:active} 2024-04-27T04:01:28.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:28.785+0000 7f31ab640700 1 -- 172.21.15.46:0/470115213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f31a402cde0 con 0x7f31a4101e30 2024-04-27T04:01:29.151 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:28 smithi046 ceph-mon[26899]: mds.? [v2:172.21.15.46:6826/2151398550,v1:172.21.15.46:6827/2151398550] up:active 2024-04-27T04:01:29.151 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:28 smithi046 ceph-mon[26899]: fsmap cephfs:1 {0=cephfs.smithi046.qgmaql=up:active} 2024-04-27T04:01:29.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:29.682+0000 7f319a7fc700 1 -- 172.21.15.46:0/470115213 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f319405a210 con 0x7f31a4101e30 2024-04-27T04:01:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:29.686+0000 7f31ab640700 1 -- 172.21.15.46:0/470115213 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f319006cce0 msgr2=0x7f319006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:29.686+0000 7f31ab640700 1 --2- 172.21.15.46:0/470115213 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f319006cce0 0x7f319006f1a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f31a000b800 tx=0x7f31a0009f90 comp rx=0 tx=0).stop 2024-04-27T04:01:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:29.686+0000 7f31ab640700 1 -- 172.21.15.46:0/470115213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31a4101e30 msgr2=0x7f31a4196b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:29.686+0000 7f31ab640700 1 --2- 172.21.15.46:0/470115213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31a4101e30 0x7f31a4196b60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f319400e920 tx=0x7f319400ece0 comp rx=0 tx=0).stop 2024-04-27T04:01:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:29.686+0000 7f31ab640700 1 -- 172.21.15.46:0/470115213 shutdown_connections 2024-04-27T04:01:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:29.686+0000 7f31ab640700 1 --2- 172.21.15.46:0/470115213 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f319006cce0 0x7f319006f1a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:29.686+0000 7f31ab640700 1 --2- 172.21.15.46:0/470115213 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31a4100c30 0x7f31a4196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:29.686+0000 7f31ab640700 1 --2- 172.21.15.46:0/470115213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31a4101e30 0x7f31a4196b60 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:29.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:29.686+0000 7f31ab640700 1 -- 172.21.15.46:0/470115213 >> 172.21.15.46:0/470115213 conn(0x7f31a40fc1d0 msgr2=0x7f31a4105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:29.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:29.686+0000 7f31ab640700 1 -- 172.21.15.46:0/470115213 shutdown_connections 2024-04-27T04:01:29.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:29.686+0000 7f31ab640700 1 -- 172.21.15.46:0/470115213 wait complete. 2024-04-27T04:01:29.705 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:29 smithi046 ceph-mon[26899]: pgmap v182: 65 pgs: 65 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 289 B/s wr, 0 op/s 2024-04-27T04:01:29.706 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:29 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/470115213' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-27T04:01:29.706 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:29 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:29.706 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:29 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:29.706 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:29 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:29.706 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:29 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi046.mycakd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T04:01:29.706 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:29 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi046.mycakd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T04:01:29.706 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:29 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:01:29.706 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:29 smithi046 ceph-mon[26899]: Deploying daemon mds.cephfs.smithi046.mycakd on smithi046 2024-04-27T04:01:30.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:29 smithi073 ceph-mon[34160]: pgmap v182: 65 pgs: 65 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 289 B/s wr, 0 op/s 2024-04-27T04:01:30.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:29 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/470115213' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-27T04:01:30.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:29 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:30.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:29 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:30.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:29 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:30.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:29 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi046.mycakd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T04:01:30.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:29 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi046.mycakd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T04:01:30.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:29 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:01:30.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:29 smithi073 ceph-mon[34160]: Deploying daemon mds.cephfs.smithi046.mycakd on smithi046 2024-04-27T04:01:30.702 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T04:01:30.711 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi046.front.sepia.ceph.com 2024-04-27T04:01:30.712 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-04-27T04:01:30.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:30 smithi046 ceph-mon[26899]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-27T04:01:30.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:30 smithi046 ceph-mon[26899]: mds.? [v2:172.21.15.73:6824/3397960510,v1:172.21.15.73:6825/3397960510] up:boot 2024-04-27T04:01:30.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:30 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/470115213' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-27T04:01:30.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:30 smithi046 ceph-mon[26899]: daemon mds.cephfs.smithi073.zkoypj assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-27T04:01:30.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:30 smithi046 ceph-mon[26899]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-27T04:01:30.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:30 smithi046 ceph-mon[26899]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-27T04:01:30.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:30 smithi046 ceph-mon[26899]: fsmap cephfs:1 {0=cephfs.smithi046.qgmaql=up:active} 1 up:standby 2024-04-27T04:01:30.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:30 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi073.zkoypj"}]: dispatch 2024-04-27T04:01:30.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:30 smithi046 ceph-mon[26899]: fsmap cephfs:2 {0=cephfs.smithi046.qgmaql=up:active,1=cephfs.smithi073.zkoypj=up:creating} 2024-04-27T04:01:30.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:30 smithi046 ceph-mon[26899]: daemon mds.cephfs.smithi073.zkoypj is now active in filesystem cephfs as rank 1 2024-04-27T04:01:31.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:30 smithi073 ceph-mon[34160]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-27T04:01:31.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:30 smithi073 ceph-mon[34160]: mds.? [v2:172.21.15.73:6824/3397960510,v1:172.21.15.73:6825/3397960510] up:boot 2024-04-27T04:01:31.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:30 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/470115213' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-27T04:01:31.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:30 smithi073 ceph-mon[34160]: daemon mds.cephfs.smithi073.zkoypj assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-27T04:01:31.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:30 smithi073 ceph-mon[34160]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-27T04:01:31.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:30 smithi073 ceph-mon[34160]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-27T04:01:31.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:30 smithi073 ceph-mon[34160]: fsmap cephfs:1 {0=cephfs.smithi046.qgmaql=up:active} 1 up:standby 2024-04-27T04:01:31.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:30 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi073.zkoypj"}]: dispatch 2024-04-27T04:01:31.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:30 smithi073 ceph-mon[34160]: fsmap cephfs:2 {0=cephfs.smithi046.qgmaql=up:active,1=cephfs.smithi073.zkoypj=up:creating} 2024-04-27T04:01:31.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:30 smithi073 ceph-mon[34160]: daemon mds.cephfs.smithi073.zkoypj is now active in filesystem cephfs as rank 1 2024-04-27T04:01:31.271 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:01:31.952 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:31 smithi046 ceph-mon[26899]: mds.? [v2:172.21.15.73:6824/3397960510,v1:172.21.15.73:6825/3397960510] up:active 2024-04-27T04:01:31.952 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:31 smithi046 ceph-mon[26899]: fsmap cephfs:2 {0=cephfs.smithi046.qgmaql=up:active,1=cephfs.smithi073.zkoypj=up:active} 2024-04-27T04:01:31.952 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:31 smithi046 ceph-mon[26899]: pgmap v183: 65 pgs: 65 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-27T04:01:31.952 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:31.952 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:01:32.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:31 smithi073 ceph-mon[34160]: mds.? [v2:172.21.15.73:6824/3397960510,v1:172.21.15.73:6825/3397960510] up:active 2024-04-27T04:01:32.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:31 smithi073 ceph-mon[34160]: fsmap cephfs:2 {0=cephfs.smithi046.qgmaql=up:active,1=cephfs.smithi073.zkoypj=up:active} 2024-04-27T04:01:32.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:31 smithi073 ceph-mon[34160]: pgmap v183: 65 pgs: 65 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-27T04:01:32.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:31 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:32.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:31 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:01:33.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.068+0000 7fb59a989700 1 -- 172.21.15.46:0/1550552606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb594100d10 msgr2=0x7fb594101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:33.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.068+0000 7fb59a989700 1 --2- 172.21.15.46:0/1550552606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb594100d10 0x7fb594101170 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb588009910 tx=0x7fb588009c20 comp rx=0 tx=0).stop 2024-04-27T04:01:33.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.068+0000 7fb59a989700 1 -- 172.21.15.46:0/1550552606 shutdown_connections 2024-04-27T04:01:33.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.068+0000 7fb59a989700 1 --2- 172.21.15.46:0/1550552606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb594100d10 0x7fb594101170 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:33.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.068+0000 7fb59a989700 1 --2- 172.21.15.46:0/1550552606 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5940ffb10 0x7fb5940fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:33.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.068+0000 7fb59a989700 1 -- 172.21.15.46:0/1550552606 >> 172.21.15.46:0/1550552606 conn(0x7fb5940fb0b0 msgr2=0x7fb5940fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:33.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.068+0000 7fb59a989700 1 -- 172.21.15.46:0/1550552606 shutdown_connections 2024-04-27T04:01:33.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.068+0000 7fb59a989700 1 -- 172.21.15.46:0/1550552606 wait complete. 2024-04-27T04:01:33.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.068+0000 7fb59a989700 1 Processor -- start 2024-04-27T04:01:33.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.068+0000 7fb59a989700 1 -- start start 2024-04-27T04:01:33.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.069+0000 7fb59a989700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5940ffb10 0x7fb594103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:33.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.069+0000 7fb59a989700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb594100d10 0x7fb594102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:33.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.069+0000 7fb59a989700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5941029b0 con 0x7fb5940ffb10 2024-04-27T04:01:33.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.069+0000 7fb59a989700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb594102af0 con 0x7fb594100d10 2024-04-27T04:01:33.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.069+0000 7fb593fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5940ffb10 0x7fb594103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:33.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.069+0000 7fb593fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5940ffb10 0x7fb594103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54280/0 (socket says 172.21.15.46:54280) 2024-04-27T04:01:33.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.069+0000 7fb593fff700 1 -- 172.21.15.46:0/4039256087 learned_addr learned my addr 172.21.15.46:0/4039256087 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:33.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.069+0000 7fb5937fe700 1 --2- 172.21.15.46:0/4039256087 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb594100d10 0x7fb594102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:33.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.069+0000 7fb593fff700 1 -- 172.21.15.46:0/4039256087 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb594100d10 msgr2=0x7fb594102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:33.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.069+0000 7fb593fff700 1 --2- 172.21.15.46:0/4039256087 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb594100d10 0x7fb594102470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:33.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.069+0000 7fb593fff700 1 -- 172.21.15.46:0/4039256087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5880095f0 con 0x7fb5940ffb10 2024-04-27T04:01:33.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.069+0000 7fb5937fe700 1 --2- 172.21.15.46:0/4039256087 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb594100d10 0x7fb594102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T04:01:33.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.069+0000 7fb593fff700 1 --2- 172.21.15.46:0/4039256087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5940ffb10 0x7fb594103dd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb58400e8f0 tx=0x7fb58400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:33.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.069+0000 7fb5917fa700 1 -- 172.21.15.46:0/4039256087 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb584009790 con 0x7fb5940ffb10 2024-04-27T04:01:33.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.069+0000 7fb59a989700 1 -- 172.21.15.46:0/4039256087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb594102dd0 con 0x7fb5940ffb10 2024-04-27T04:01:33.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.069+0000 7fb5917fa700 1 -- 172.21.15.46:0/4039256087 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5840044e0 con 0x7fb5940ffb10 2024-04-27T04:01:33.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.070+0000 7fb5917fa700 1 -- 172.21.15.46:0/4039256087 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb584003e00 con 0x7fb5940ffb10 2024-04-27T04:01:33.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.070+0000 7fb59a989700 1 -- 172.21.15.46:0/4039256087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59419e630 con 0x7fb5940ffb10 2024-04-27T04:01:33.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.071+0000 7fb59a989700 1 -- 172.21.15.46:0/4039256087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb59404ebb0 con 0x7fb5940ffb10 2024-04-27T04:01:33.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.071+0000 7fb5917fa700 1 -- 172.21.15.46:0/4039256087 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb58400cba0 con 0x7fb5940ffb10 2024-04-27T04:01:33.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.071+0000 7fb5917fa700 1 --2- 172.21.15.46:0/4039256087 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb57c06cc80 0x7fb57c06f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:33.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.072+0000 7fb5917fa700 1 -- 172.21.15.46:0/4039256087 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fb584014070 con 0x7fb5940ffb10 2024-04-27T04:01:33.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.072+0000 7fb5937fe700 1 --2- 172.21.15.46:0/4039256087 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb57c06cc80 0x7fb57c06f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:33.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.072+0000 7fb5937fe700 1 --2- 172.21.15.46:0/4039256087 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb57c06cc80 0x7fb57c06f140 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb5880059d0 tx=0x7fb588005940 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:33.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.076+0000 7fb5917fa700 1 -- 172.21.15.46:0/4039256087 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fb58405a4f0 con 0x7fb5940ffb10 2024-04-27T04:01:33.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.337+0000 7fb59a989700 1 -- 172.21.15.46:0/4039256087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fb594108670 con 0x7fb5940ffb10 2024-04-27T04:01:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.833+0000 7fb5917fa700 1 -- 172.21.15.46:0/4039256087 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v9) v1 ==== 121+0+0 (secure 0 0 0) 0x7fb58405a080 con 0x7fb5940ffb10 2024-04-27T04:01:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.836+0000 7fb59a989700 1 -- 172.21.15.46:0/4039256087 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb57c06cc80 msgr2=0x7fb57c06f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.836+0000 7fb59a989700 1 --2- 172.21.15.46:0/4039256087 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb57c06cc80 0x7fb57c06f140 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb5880059d0 tx=0x7fb588005940 comp rx=0 tx=0).stop 2024-04-27T04:01:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.836+0000 7fb59a989700 1 -- 172.21.15.46:0/4039256087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5940ffb10 msgr2=0x7fb594103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.836+0000 7fb59a989700 1 --2- 172.21.15.46:0/4039256087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5940ffb10 0x7fb594103dd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb58400e8f0 tx=0x7fb58400ecb0 comp rx=0 tx=0).stop 2024-04-27T04:01:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.836+0000 7fb59a989700 1 -- 172.21.15.46:0/4039256087 shutdown_connections 2024-04-27T04:01:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.836+0000 7fb59a989700 1 --2- 172.21.15.46:0/4039256087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5940ffb10 0x7fb594103dd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.836+0000 7fb59a989700 1 --2- 172.21.15.46:0/4039256087 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb57c06cc80 0x7fb57c06f140 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.836+0000 7fb59a989700 1 --2- 172.21.15.46:0/4039256087 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb594100d10 0x7fb594102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.836+0000 7fb59a989700 1 -- 172.21.15.46:0/4039256087 >> 172.21.15.46:0/4039256087 conn(0x7fb5940fb0b0 msgr2=0x7fb594068e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.836+0000 7fb59a989700 1 -- 172.21.15.46:0/4039256087 shutdown_connections 2024-04-27T04:01:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:33.836+0000 7fb59a989700 1 -- 172.21.15.46:0/4039256087 wait complete. 2024-04-27T04:01:34.150 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:33 smithi046 ceph-mon[26899]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s wr, 6 op/s 2024-04-27T04:01:34.150 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:33 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/4039256087' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-27T04:01:34.150 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:33 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:34.150 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:33 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:34.150 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:33 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:34.151 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:33 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi073.yryono", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T04:01:34.151 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:33 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi073.yryono", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T04:01:34.151 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:33 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:01:34.151 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:33 smithi046 ceph-mon[26899]: Deploying daemon mds.cephfs.smithi073.yryono on smithi073 2024-04-27T04:01:34.180 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:33 smithi073 ceph-mon[34160]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s wr, 6 op/s 2024-04-27T04:01:34.180 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:33 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/4039256087' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-27T04:01:34.180 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:33 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:34.180 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:33 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:34.180 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:33 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:34.180 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:33 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi073.yryono", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T04:01:34.180 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:33 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi073.yryono", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T04:01:34.180 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:33 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:01:34.180 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:33 smithi073 ceph-mon[34160]: Deploying daemon mds.cephfs.smithi073.yryono on smithi073 2024-04-27T04:01:34.455 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T04:01:34.466 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi046.front.sepia.ceph.com 2024-04-27T04:01:34.466 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-04-27T04:01:34.680 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:01:35.112 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:34 smithi073 ceph-mon[34160]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-27T04:01:35.112 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:34 smithi073 ceph-mon[34160]: Cluster is now healthy 2024-04-27T04:01:35.112 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:34 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/4039256087' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-27T04:01:35.112 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:34 smithi073 ceph-mon[34160]: mds.? [v2:172.21.15.46:6828/3081610305,v1:172.21.15.46:6829/3081610305] up:boot 2024-04-27T04:01:35.113 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:34 smithi073 ceph-mon[34160]: fsmap cephfs:2 {0=cephfs.smithi046.qgmaql=up:active,1=cephfs.smithi073.zkoypj=up:active} 1 up:standby 2024-04-27T04:01:35.113 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:34 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi046.mycakd"}]: dispatch 2024-04-27T04:01:35.113 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:34 smithi073 ceph-mon[34160]: fsmap cephfs:2 {0=cephfs.smithi046.qgmaql=up:active,1=cephfs.smithi073.zkoypj=up:active} 1 up:standby-replay 2024-04-27T04:01:35.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:34 smithi046 ceph-mon[26899]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-27T04:01:35.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:34 smithi046 ceph-mon[26899]: Cluster is now healthy 2024-04-27T04:01:35.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:34 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/4039256087' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-27T04:01:35.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:34 smithi046 ceph-mon[26899]: mds.? [v2:172.21.15.46:6828/3081610305,v1:172.21.15.46:6829/3081610305] up:boot 2024-04-27T04:01:35.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:34 smithi046 ceph-mon[26899]: fsmap cephfs:2 {0=cephfs.smithi046.qgmaql=up:active,1=cephfs.smithi073.zkoypj=up:active} 1 up:standby 2024-04-27T04:01:35.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:34 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi046.mycakd"}]: dispatch 2024-04-27T04:01:35.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:34 smithi046 ceph-mon[26899]: fsmap cephfs:2 {0=cephfs.smithi046.qgmaql=up:active,1=cephfs.smithi073.zkoypj=up:active} 1 up:standby-replay 2024-04-27T04:01:35.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:35 smithi046 ceph-mon[26899]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s wr, 7 op/s 2024-04-27T04:01:35.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:35 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:36.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:35 smithi073 ceph-mon[34160]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s wr, 7 op/s 2024-04-27T04:01:36.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:35 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:36.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.489+0000 7f18945cb700 1 -- 172.21.15.46:0/1125009718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c101710 msgr2=0x7f188c103b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:36.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.489+0000 7f18945cb700 1 --2- 172.21.15.46:0/1125009718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c101710 0x7f188c103b00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1880009910 tx=0x7f1880009c20 comp rx=0 tx=0).stop 2024-04-27T04:01:36.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.489+0000 7f18945cb700 1 -- 172.21.15.46:0/1125009718 shutdown_connections 2024-04-27T04:01:36.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.489+0000 7f18945cb700 1 --2- 172.21.15.46:0/1125009718 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f188c104040 0x7f188c106430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:36.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.489+0000 7f18945cb700 1 --2- 172.21.15.46:0/1125009718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c101710 0x7f188c103b00 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:36.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.489+0000 7f18945cb700 1 -- 172.21.15.46:0/1125009718 >> 172.21.15.46:0/1125009718 conn(0x7f188c0fb0c0 msgr2=0x7f188c0fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:36.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.489+0000 7f18945cb700 1 -- 172.21.15.46:0/1125009718 shutdown_connections 2024-04-27T04:01:36.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.490+0000 7f18945cb700 1 -- 172.21.15.46:0/1125009718 wait complete. 2024-04-27T04:01:36.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.490+0000 7f18945cb700 1 Processor -- start 2024-04-27T04:01:36.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.490+0000 7f18945cb700 1 -- start start 2024-04-27T04:01:36.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.490+0000 7f18945cb700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f188c101710 0x7f188c196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:36.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.490+0000 7f18945cb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c104040 0x7f188c196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:36.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.490+0000 7f18945cb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188c1971c0 con 0x7f188c104040 2024-04-27T04:01:36.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.490+0000 7f18945cb700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188c197300 con 0x7f188c101710 2024-04-27T04:01:36.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.491+0000 7f1892367700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f188c101710 0x7f188c196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:36.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.491+0000 7f1891b66700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c104040 0x7f188c196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:36.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.491+0000 7f1891b66700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c104040 0x7f188c196ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54310/0 (socket says 172.21.15.46:54310) 2024-04-27T04:01:36.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.491+0000 7f1892367700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f188c101710 0x7f188c196660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.46:57184/0 (socket says 172.21.15.46:57184) 2024-04-27T04:01:36.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.491+0000 7f1891b66700 1 -- 172.21.15.46:0/2280546510 learned_addr learned my addr 172.21.15.46:0/2280546510 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:36.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.491+0000 7f1891b66700 1 -- 172.21.15.46:0/2280546510 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f188c101710 msgr2=0x7f188c196660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:36.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.491+0000 7f1891b66700 1 --2- 172.21.15.46:0/2280546510 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f188c101710 0x7f188c196660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:36.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.491+0000 7f1891b66700 1 -- 172.21.15.46:0/2280546510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18800095f0 con 0x7f188c104040 2024-04-27T04:01:36.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.491+0000 7f1891b66700 1 --2- 172.21.15.46:0/2280546510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c104040 0x7f188c196ba0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f188800e940 tx=0x7f188800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:36.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.491+0000 7f187f7fe700 1 -- 172.21.15.46:0/2280546510 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f188800cba0 con 0x7f188c104040 2024-04-27T04:01:36.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.491+0000 7f18945cb700 1 -- 172.21.15.46:0/2280546510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f188c0ff560 con 0x7f188c104040 2024-04-27T04:01:36.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.491+0000 7f187f7fe700 1 -- 172.21.15.46:0/2280546510 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f188800ee90 con 0x7f188c104040 2024-04-27T04:01:36.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.491+0000 7f187f7fe700 1 -- 172.21.15.46:0/2280546510 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18880055d0 con 0x7f188c104040 2024-04-27T04:01:36.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.491+0000 7f18945cb700 1 -- 172.21.15.46:0/2280546510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f188c0ffb30 con 0x7f188c104040 2024-04-27T04:01:36.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.493+0000 7f18945cb700 1 -- 172.21.15.46:0/2280546510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f188c04ebb0 con 0x7f188c104040 2024-04-27T04:01:36.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.493+0000 7f187f7fe700 1 -- 172.21.15.46:0/2280546510 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1888004750 con 0x7f188c104040 2024-04-27T04:01:36.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.493+0000 7f187f7fe700 1 --2- 172.21.15.46:0/2280546510 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f187806cce0 0x7f187806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:36.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.493+0000 7f187f7fe700 1 -- 172.21.15.46:0/2280546510 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f1888014070 con 0x7f188c104040 2024-04-27T04:01:36.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.493+0000 7f1892367700 1 --2- 172.21.15.46:0/2280546510 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f187806cce0 0x7f187806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:36.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.494+0000 7f1892367700 1 --2- 172.21.15.46:0/2280546510 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f187806cce0 0x7f187806f1a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1880006010 tx=0x7f1880009f70 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:36.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.498+0000 7f187f7fe700 1 -- 172.21.15.46:0/2280546510 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f188805a3d0 con 0x7f188c104040 2024-04-27T04:01:36.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.778+0000 7f18945cb700 1 -- 172.21.15.46:0/2280546510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f188c1097a0 con 0x7f188c104040 2024-04-27T04:01:36.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.843+0000 7f187f7fe700 1 -- 172.21.15.46:0/2280546510 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v11) v1 ==== 168+0+0 (secure 0 0 0) 0x7f1888059f60 con 0x7f188c104040 2024-04-27T04:01:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.845+0000 7f18945cb700 1 -- 172.21.15.46:0/2280546510 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f187806cce0 msgr2=0x7f187806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.845+0000 7f18945cb700 1 --2- 172.21.15.46:0/2280546510 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f187806cce0 0x7f187806f1a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1880006010 tx=0x7f1880009f70 comp rx=0 tx=0).stop 2024-04-27T04:01:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.845+0000 7f18945cb700 1 -- 172.21.15.46:0/2280546510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c104040 msgr2=0x7f188c196ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.845+0000 7f18945cb700 1 --2- 172.21.15.46:0/2280546510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c104040 0x7f188c196ba0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f188800e940 tx=0x7f188800ec50 comp rx=0 tx=0).stop 2024-04-27T04:01:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.845+0000 7f18945cb700 1 -- 172.21.15.46:0/2280546510 shutdown_connections 2024-04-27T04:01:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.845+0000 7f18945cb700 1 --2- 172.21.15.46:0/2280546510 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f187806cce0 0x7f187806f1a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.845+0000 7f18945cb700 1 --2- 172.21.15.46:0/2280546510 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f188c101710 0x7f188c196660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.845+0000 7f18945cb700 1 --2- 172.21.15.46:0/2280546510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c104040 0x7f188c196ba0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.845+0000 7f18945cb700 1 -- 172.21.15.46:0/2280546510 >> 172.21.15.46:0/2280546510 conn(0x7f188c0fb0c0 msgr2=0x7f188c0fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.845+0000 7f18945cb700 1 -- 172.21.15.46:0/2280546510 shutdown_connections 2024-04-27T04:01:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:36.845+0000 7f18945cb700 1 -- 172.21.15.46:0/2280546510 wait complete. 2024-04-27T04:01:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:inline data enabled 2024-04-27T04:01:37.283 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:37 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:37.283 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:37 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:37.283 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:37 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:37.283 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:37 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:37.283 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:37 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:37.284 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:37 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:01:37.284 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:37 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/2280546510' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T04:01:37.284 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:37 smithi073 ceph-mon[34160]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-27T04:01:37.284 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:37 smithi073 ceph-mon[34160]: mds.? [v2:172.21.15.73:6826/565262827,v1:172.21.15.73:6827/565262827] up:boot 2024-04-27T04:01:37.284 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:37 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/2280546510' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T04:01:37.284 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:37 smithi073 ceph-mon[34160]: fsmap cephfs:2 {0=cephfs.smithi046.qgmaql=up:active,1=cephfs.smithi073.zkoypj=up:active} 1 up:standby-replay 1 up:standby 2024-04-27T04:01:37.284 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:37 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi073.yryono"}]: dispatch 2024-04-27T04:01:37.284 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:37 smithi073 ceph-mon[34160]: fsmap cephfs:2 {0=cephfs.smithi046.qgmaql=up:active,1=cephfs.smithi073.zkoypj=up:active} 2 up:standby-replay 2024-04-27T04:01:37.387 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:37.387 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:37.387 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:37.387 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:37.388 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:37.388 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:01:37.388 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:37 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2280546510' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T04:01:37.388 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:37 smithi046 ceph-mon[26899]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-27T04:01:37.388 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:37 smithi046 ceph-mon[26899]: mds.? [v2:172.21.15.73:6826/565262827,v1:172.21.15.73:6827/565262827] up:boot 2024-04-27T04:01:37.388 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:37 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2280546510' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T04:01:37.388 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:37 smithi046 ceph-mon[26899]: fsmap cephfs:2 {0=cephfs.smithi046.qgmaql=up:active,1=cephfs.smithi073.zkoypj=up:active} 1 up:standby-replay 1 up:standby 2024-04-27T04:01:37.388 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:37 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi073.yryono"}]: dispatch 2024-04-27T04:01:37.388 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:37 smithi046 ceph-mon[26899]: fsmap cephfs:2 {0=cephfs.smithi046.qgmaql=up:active,1=cephfs.smithi073.zkoypj=up:active} 2 up:standby-replay 2024-04-27T04:01:37.990 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T04:01:38.002 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi046.front.sepia.ceph.com 2024-04-27T04:01:38.002 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-27T04:01:38.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:38 smithi073 ceph-mon[34160]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-27T04:01:38.356 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:38 smithi046 ceph-mon[26899]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-27T04:01:38.886 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:01:39.316 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:39 smithi046 ceph-mon[26899]: mds.? [v2:172.21.15.46:6828/3081610305,v1:172.21.15.46:6829/3081610305] up:standby-replay 2024-04-27T04:01:39.316 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:39 smithi046 ceph-mon[26899]: mds.? [v2:172.21.15.73:6824/3397960510,v1:172.21.15.73:6825/3397960510] up:active 2024-04-27T04:01:39.316 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:39 smithi046 ceph-mon[26899]: Dropping low affinity active daemon mds.cephfs.smithi046.qgmaql in favor of higher affinity standby. 2024-04-27T04:01:39.316 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:39 smithi046 ceph-mon[26899]: Replacing daemon mds.cephfs.smithi046.qgmaql as rank 0 with standby daemon mds.cephfs.smithi046.mycakd 2024-04-27T04:01:39.317 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:39 smithi046 ceph-mon[26899]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T04:01:39.317 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:39 smithi046 ceph-mon[26899]: fsmap cephfs:2 {0=cephfs.smithi046.qgmaql=up:active,1=cephfs.smithi073.zkoypj=up:active} 2 up:standby-replay 2024-04-27T04:01:39.317 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:39 smithi046 ceph-mon[26899]: osdmap e46: 6 total, 6 up, 6 in 2024-04-27T04:01:39.317 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:39 smithi046 ceph-mon[26899]: fsmap cephfs:2/2 {0=cephfs.smithi046.mycakd=up:replay,1=cephfs.smithi073.zkoypj=up:active} 1 up:standby-replay 2024-04-27T04:01:39.317 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:39.317 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:39.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:39 smithi073 ceph-mon[34160]: mds.? [v2:172.21.15.46:6828/3081610305,v1:172.21.15.46:6829/3081610305] up:standby-replay 2024-04-27T04:01:39.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:39 smithi073 ceph-mon[34160]: mds.? [v2:172.21.15.73:6824/3397960510,v1:172.21.15.73:6825/3397960510] up:active 2024-04-27T04:01:39.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:39 smithi073 ceph-mon[34160]: Dropping low affinity active daemon mds.cephfs.smithi046.qgmaql in favor of higher affinity standby. 2024-04-27T04:01:39.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:39 smithi073 ceph-mon[34160]: Replacing daemon mds.cephfs.smithi046.qgmaql as rank 0 with standby daemon mds.cephfs.smithi046.mycakd 2024-04-27T04:01:39.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:39 smithi073 ceph-mon[34160]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T04:01:39.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:39 smithi073 ceph-mon[34160]: fsmap cephfs:2 {0=cephfs.smithi046.qgmaql=up:active,1=cephfs.smithi073.zkoypj=up:active} 2 up:standby-replay 2024-04-27T04:01:39.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:39 smithi073 ceph-mon[34160]: osdmap e46: 6 total, 6 up, 6 in 2024-04-27T04:01:39.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:39 smithi073 ceph-mon[34160]: fsmap cephfs:2/2 {0=cephfs.smithi046.mycakd=up:replay,1=cephfs.smithi073.zkoypj=up:active} 1 up:standby-replay 2024-04-27T04:01:39.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:39.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:40.348 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:40 smithi046 ceph-mon[26899]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-04-27T04:01:40.348 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:40 smithi046 ceph-mon[26899]: mds.? [v2:172.21.15.46:6826/2653255940,v1:172.21.15.46:6827/2653255940] up:boot 2024-04-27T04:01:40.348 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:40 smithi046 ceph-mon[26899]: mds.? [v2:172.21.15.46:6828/3081610305,v1:172.21.15.46:6829/3081610305] up:resolve 2024-04-27T04:01:40.348 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:40 smithi046 ceph-mon[26899]: fsmap cephfs:2/2 {0=cephfs.smithi046.mycakd=up:resolve,1=cephfs.smithi073.zkoypj=up:active} 1 up:standby-replay 1 up:standby 2024-04-27T04:01:40.349 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:40 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi046.qgmaql"}]: dispatch 2024-04-27T04:01:40.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:40 smithi073 ceph-mon[34160]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-04-27T04:01:40.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:40 smithi073 ceph-mon[34160]: mds.? [v2:172.21.15.46:6826/2653255940,v1:172.21.15.46:6827/2653255940] up:boot 2024-04-27T04:01:40.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:40 smithi073 ceph-mon[34160]: mds.? [v2:172.21.15.46:6828/3081610305,v1:172.21.15.46:6829/3081610305] up:resolve 2024-04-27T04:01:40.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:40 smithi073 ceph-mon[34160]: fsmap cephfs:2/2 {0=cephfs.smithi046.mycakd=up:resolve,1=cephfs.smithi073.zkoypj=up:active} 1 up:standby-replay 1 up:standby 2024-04-27T04:01:40.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:40 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi046.qgmaql"}]: dispatch 2024-04-27T04:01:41.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:41 smithi046 ceph-mon[26899]: mds.? [v2:172.21.15.46:6828/3081610305,v1:172.21.15.46:6829/3081610305] up:reconnect 2024-04-27T04:01:41.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:41 smithi046 ceph-mon[26899]: mds.? [v2:172.21.15.73:6826/565262827,v1:172.21.15.73:6827/565262827] up:standby-replay 2024-04-27T04:01:41.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:41 smithi046 ceph-mon[26899]: fsmap cephfs:2/2 {0=cephfs.smithi046.mycakd=up:reconnect,1=cephfs.smithi073.zkoypj=up:active} 1 up:standby-replay 1 up:standby 2024-04-27T04:01:41.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:41 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:41.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:41 smithi073 ceph-mon[34160]: mds.? [v2:172.21.15.46:6828/3081610305,v1:172.21.15.46:6829/3081610305] up:reconnect 2024-04-27T04:01:41.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:41 smithi073 ceph-mon[34160]: mds.? [v2:172.21.15.73:6826/565262827,v1:172.21.15.73:6827/565262827] up:standby-replay 2024-04-27T04:01:41.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:41 smithi073 ceph-mon[34160]: fsmap cephfs:2/2 {0=cephfs.smithi046.mycakd=up:reconnect,1=cephfs.smithi073.zkoypj=up:active} 1 up:standby-replay 1 up:standby 2024-04-27T04:01:41.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:41 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:41.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.708+0000 7f1119d96700 1 -- 172.21.15.46:0/3892778409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1114101e10 msgr2=0x7f1114102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:41.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.708+0000 7f1119d96700 1 --2- 172.21.15.46:0/3892778409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1114101e10 0x7f1114102270 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009910 tx=0x7f10fc009c20 comp rx=0 tx=0).stop 2024-04-27T04:01:41.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.708+0000 7f1119d96700 1 -- 172.21.15.46:0/3892778409 shutdown_connections 2024-04-27T04:01:41.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.708+0000 7f1119d96700 1 --2- 172.21.15.46:0/3892778409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1114101e10 0x7f1114102270 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:41.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.708+0000 7f1119d96700 1 --2- 172.21.15.46:0/3892778409 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1114100c10 0x7f1114101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:41.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.708+0000 7f1119d96700 1 -- 172.21.15.46:0/3892778409 >> 172.21.15.46:0/3892778409 conn(0x7f11140fc1f0 msgr2=0x7f11140fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:41.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.708+0000 7f1119d96700 1 -- 172.21.15.46:0/3892778409 shutdown_connections 2024-04-27T04:01:41.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.708+0000 7f1119d96700 1 -- 172.21.15.46:0/3892778409 wait complete. 2024-04-27T04:01:41.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.709+0000 7f1119d96700 1 Processor -- start 2024-04-27T04:01:41.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.709+0000 7f1119d96700 1 -- start start 2024-04-27T04:01:41.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.709+0000 7f1119d96700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1114100c10 0x7f11141967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:41.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.709+0000 7f1119d96700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1114101e10 0x7f1114196cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:41.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.709+0000 7f1119d96700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1114197310 con 0x7f1114100c10 2024-04-27T04:01:41.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.709+0000 7f1119d96700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1114197450 con 0x7f1114101e10 2024-04-27T04:01:41.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.709+0000 7f11137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1114100c10 0x7f11141967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:41.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.709+0000 7f11137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1114100c10 0x7f11141967b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51078/0 (socket says 172.21.15.46:51078) 2024-04-27T04:01:41.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.709+0000 7f11137fe700 1 -- 172.21.15.46:0/3649643884 learned_addr learned my addr 172.21.15.46:0/3649643884 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:41.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.709+0000 7f1112ffd700 1 --2- 172.21.15.46:0/3649643884 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1114101e10 0x7f1114196cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:41.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.709+0000 7f11137fe700 1 -- 172.21.15.46:0/3649643884 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1114101e10 msgr2=0x7f1114196cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:41.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.709+0000 7f11137fe700 1 --2- 172.21.15.46:0/3649643884 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1114101e10 0x7f1114196cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:41.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.709+0000 7f11137fe700 1 -- 172.21.15.46:0/3649643884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10fc0095f0 con 0x7f1114100c10 2024-04-27T04:01:41.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.710+0000 7f11137fe700 1 --2- 172.21.15.46:0/3649643884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1114100c10 0x7f11141967b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f110c00b9c0 tx=0x7f110c00bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:41.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.710+0000 7f1110ff9700 1 -- 172.21.15.46:0/3649643884 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f110c00c980 con 0x7f1114100c10 2024-04-27T04:01:41.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.710+0000 7f1119d96700 1 -- 172.21.15.46:0/3649643884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f111419bdf0 con 0x7f1114100c10 2024-04-27T04:01:41.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.710+0000 7f1110ff9700 1 -- 172.21.15.46:0/3649643884 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f110c014440 con 0x7f1114100c10 2024-04-27T04:01:41.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.710+0000 7f1110ff9700 1 -- 172.21.15.46:0/3649643884 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f110c012570 con 0x7f1114100c10 2024-04-27T04:01:41.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.710+0000 7f1119d96700 1 -- 172.21.15.46:0/3649643884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f111419c340 con 0x7f1114100c10 2024-04-27T04:01:41.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.711+0000 7f1119d96700 1 -- 172.21.15.46:0/3649643884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1114190830 con 0x7f1114100c10 2024-04-27T04:01:41.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.711+0000 7f1110ff9700 1 -- 172.21.15.46:0/3649643884 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f110c00cae0 con 0x7f1114100c10 2024-04-27T04:01:41.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.712+0000 7f1110ff9700 1 --2- 172.21.15.46:0/3649643884 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f110006cce0 0x7f110006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:41.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.712+0000 7f1110ff9700 1 -- 172.21.15.46:0/3649643884 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f110c006e40 con 0x7f1114100c10 2024-04-27T04:01:41.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.712+0000 7f1112ffd700 1 --2- 172.21.15.46:0/3649643884 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f110006cce0 0x7f110006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:41.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.712+0000 7f1112ffd700 1 --2- 172.21.15.46:0/3649643884 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f110006cce0 0x7f110006f1a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0051c0 tx=0x7f10fc005920 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:41.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.717+0000 7f1110ff9700 1 -- 172.21.15.46:0/3649643884 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f110c059920 con 0x7f1114100c10 2024-04-27T04:01:41.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.981+0000 7f1119d96700 1 -- 172.21.15.46:0/3649643884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f111402cd40 con 0x7f1114100c10 2024-04-27T04:01:41.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.981+0000 7f1110ff9700 1 -- 172.21.15.46:0/3649643884 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 17 v17) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f110c019770 con 0x7f1114100c10 2024-04-27T04:01:41.983 INFO:teuthology.orchestra.run.smithi046.stdout:e17 2024-04-27T04:01:41.983 INFO:teuthology.orchestra.run.smithi046.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T04:01:41.984 INFO:teuthology.orchestra.run.smithi046.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T04:01:41.984 INFO:teuthology.orchestra.run.smithi046.stdout:legacy client fscid: 1 2024-04-27T04:01:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:01:41.984 INFO:teuthology.orchestra.run.smithi046.stdout:Filesystem 'cephfs' (1) 2024-04-27T04:01:41.984 INFO:teuthology.orchestra.run.smithi046.stdout:fs_name cephfs 2024-04-27T04:01:41.984 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 17 2024-04-27T04:01:41.984 INFO:teuthology.orchestra.run.smithi046.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-27T04:01:41.984 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-04-27T04:01:21.676811+0000 2024-04-27T04:01:41.984 INFO:teuthology.orchestra.run.smithi046.stdout:modified 2024-04-27T04:01:41.120005+0000 2024-04-27T04:01:41.984 INFO:teuthology.orchestra.run.smithi046.stdout:tableserver 0 2024-04-27T04:01:41.984 INFO:teuthology.orchestra.run.smithi046.stdout:root 0 2024-04-27T04:01:41.984 INFO:teuthology.orchestra.run.smithi046.stdout:session_timeout 60 2024-04-27T04:01:41.984 INFO:teuthology.orchestra.run.smithi046.stdout:session_autoclose 300 2024-04-27T04:01:41.985 INFO:teuthology.orchestra.run.smithi046.stdout:max_file_size 1099511627776 2024-04-27T04:01:41.985 INFO:teuthology.orchestra.run.smithi046.stdout:max_xattr_size 65536 2024-04-27T04:01:41.985 INFO:teuthology.orchestra.run.smithi046.stdout:required_client_features {} 2024-04-27T04:01:41.985 INFO:teuthology.orchestra.run.smithi046.stdout:last_failure 0 2024-04-27T04:01:41.985 INFO:teuthology.orchestra.run.smithi046.stdout:last_failure_osd_epoch 46 2024-04-27T04:01:41.985 INFO:teuthology.orchestra.run.smithi046.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T04:01:41.985 INFO:teuthology.orchestra.run.smithi046.stdout:max_mds 2 2024-04-27T04:01:41.985 INFO:teuthology.orchestra.run.smithi046.stdout:in 0,1 2024-04-27T04:01:41.985 INFO:teuthology.orchestra.run.smithi046.stdout:up {0=14488,1=24285} 2024-04-27T04:01:41.985 INFO:teuthology.orchestra.run.smithi046.stdout:failed 2024-04-27T04:01:41.985 INFO:teuthology.orchestra.run.smithi046.stdout:damaged 2024-04-27T04:01:41.985 INFO:teuthology.orchestra.run.smithi046.stdout:stopped 2024-04-27T04:01:41.986 INFO:teuthology.orchestra.run.smithi046.stdout:data_pools [3] 2024-04-27T04:01:41.986 INFO:teuthology.orchestra.run.smithi046.stdout:metadata_pool 2 2024-04-27T04:01:41.986 INFO:teuthology.orchestra.run.smithi046.stdout:inline_data enabled 2024-04-27T04:01:41.986 INFO:teuthology.orchestra.run.smithi046.stdout:balancer 2024-04-27T04:01:41.986 INFO:teuthology.orchestra.run.smithi046.stdout:bal_rank_mask -1 2024-04-27T04:01:41.986 INFO:teuthology.orchestra.run.smithi046.stdout:standby_count_wanted 1 2024-04-27T04:01:41.986 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.cephfs.smithi046.mycakd{0:14488} state up:rejoin seq 5 join_fscid=1 addr [v2:172.21.15.46:6828/3081610305,v1:172.21.15.46:6829/3081610305] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T04:01:41.986 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.cephfs.smithi073.zkoypj{1:24285} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.73:6824/3397960510,v1:172.21.15.73:6825/3397960510] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T04:01:41.986 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.cephfs.smithi073.yryono{1:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.73:6826/565262827,v1:172.21.15.73:6827/565262827] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T04:01:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:01:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:01:41.986 INFO:teuthology.orchestra.run.smithi046.stdout:Standby daemons: 2024-04-27T04:01:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:01:41.987 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.cephfs.smithi046.qgmaql{-1:24303} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.46:6826/2653255940,v1:172.21.15.46:6827/2653255940] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T04:01:41.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.983+0000 7f1119d96700 1 -- 172.21.15.46:0/3649643884 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f110006cce0 msgr2=0x7f110006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:41.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.983+0000 7f1119d96700 1 --2- 172.21.15.46:0/3649643884 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f110006cce0 0x7f110006f1a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0051c0 tx=0x7f10fc005920 comp rx=0 tx=0).stop 2024-04-27T04:01:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.983+0000 7f1119d96700 1 -- 172.21.15.46:0/3649643884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1114100c10 msgr2=0x7f11141967b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.984+0000 7f1119d96700 1 --2- 172.21.15.46:0/3649643884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1114100c10 0x7f11141967b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f110c00b9c0 tx=0x7f110c00bd80 comp rx=0 tx=0).stop 2024-04-27T04:01:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.984+0000 7f1119d96700 1 -- 172.21.15.46:0/3649643884 shutdown_connections 2024-04-27T04:01:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.984+0000 7f1119d96700 1 --2- 172.21.15.46:0/3649643884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1114100c10 0x7f11141967b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.984+0000 7f1119d96700 1 --2- 172.21.15.46:0/3649643884 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f110006cce0 0x7f110006f1a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.984+0000 7f1119d96700 1 --2- 172.21.15.46:0/3649643884 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1114101e10 0x7f1114196cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.984+0000 7f1119d96700 1 -- 172.21.15.46:0/3649643884 >> 172.21.15.46:0/3649643884 conn(0x7f11140fc1f0 msgr2=0x7f1114105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.984+0000 7f1119d96700 1 -- 172.21.15.46:0/3649643884 shutdown_connections 2024-04-27T04:01:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:41.984+0000 7f1119d96700 1 -- 172.21.15.46:0/3649643884 wait complete. 2024-04-27T04:01:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 17 2024-04-27T04:01:42.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:42 smithi046 ceph-mon[26899]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-04-27T04:01:42.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:42 smithi046 ceph-mon[26899]: mds.? [v2:172.21.15.46:6828/3081610305,v1:172.21.15.46:6829/3081610305] up:rejoin 2024-04-27T04:01:42.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:42 smithi046 ceph-mon[26899]: fsmap cephfs:2/2 {0=cephfs.smithi046.mycakd=up:rejoin,1=cephfs.smithi073.zkoypj=up:active} 1 up:standby-replay 1 up:standby 2024-04-27T04:01:42.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:42 smithi046 ceph-mon[26899]: daemon mds.cephfs.smithi046.mycakd is now active in filesystem cephfs as rank 0 2024-04-27T04:01:42.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:42 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/3649643884' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T04:01:42.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:42 smithi073 ceph-mon[34160]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-04-27T04:01:42.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:42 smithi073 ceph-mon[34160]: mds.? [v2:172.21.15.46:6828/3081610305,v1:172.21.15.46:6829/3081610305] up:rejoin 2024-04-27T04:01:42.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:42 smithi073 ceph-mon[34160]: fsmap cephfs:2/2 {0=cephfs.smithi046.mycakd=up:rejoin,1=cephfs.smithi073.zkoypj=up:active} 1 up:standby-replay 1 up:standby 2024-04-27T04:01:42.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:42 smithi073 ceph-mon[34160]: daemon mds.cephfs.smithi046.mycakd is now active in filesystem cephfs as rank 0 2024-04-27T04:01:42.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:42 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/3649643884' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T04:01:42.635 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-27T04:01:43.048 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:01:43.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:43 smithi046 ceph-mon[26899]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T04:01:43.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:43 smithi046 ceph-mon[26899]: mds.? [v2:172.21.15.46:6828/3081610305,v1:172.21.15.46:6829/3081610305] up:active 2024-04-27T04:01:43.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:43 smithi046 ceph-mon[26899]: fsmap cephfs:2 {0=cephfs.smithi046.mycakd=up:active,1=cephfs.smithi073.zkoypj=up:active} 1 up:standby-replay 1 up:standby 2024-04-27T04:01:43.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:43 smithi046 ceph-mon[26899]: fsmap cephfs:2 {0=cephfs.smithi046.mycakd=up:active,1=cephfs.smithi073.zkoypj=up:active} 2 up:standby-replay 2024-04-27T04:01:43.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:43 smithi073 ceph-mon[34160]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T04:01:43.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:43 smithi073 ceph-mon[34160]: mds.? [v2:172.21.15.46:6828/3081610305,v1:172.21.15.46:6829/3081610305] up:active 2024-04-27T04:01:43.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:43 smithi073 ceph-mon[34160]: fsmap cephfs:2 {0=cephfs.smithi046.mycakd=up:active,1=cephfs.smithi073.zkoypj=up:active} 1 up:standby-replay 1 up:standby 2024-04-27T04:01:43.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:43 smithi073 ceph-mon[34160]: fsmap cephfs:2 {0=cephfs.smithi046.mycakd=up:active,1=cephfs.smithi073.zkoypj=up:active} 2 up:standby-replay 2024-04-27T04:01:44.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:44 smithi046 ceph-mon[26899]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-04-27T04:01:44.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:44 smithi073 ceph-mon[34160]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-04-27T04:01:44.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.739+0000 7fee46c68700 1 -- 172.21.15.46:0/3648015905 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee40100d50 msgr2=0x7fee401011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:44.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.739+0000 7fee46c68700 1 --2- 172.21.15.46:0/3648015905 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee40100d50 0x7fee401011b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009910 tx=0x7fee2c009c20 comp rx=0 tx=0).stop 2024-04-27T04:01:44.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.739+0000 7fee46c68700 1 -- 172.21.15.46:0/3648015905 shutdown_connections 2024-04-27T04:01:44.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.739+0000 7fee46c68700 1 --2- 172.21.15.46:0/3648015905 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee40100d50 0x7fee401011b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:44.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.739+0000 7fee46c68700 1 --2- 172.21.15.46:0/3648015905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee400ffb50 0x7fee400fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:44.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.739+0000 7fee46c68700 1 -- 172.21.15.46:0/3648015905 >> 172.21.15.46:0/3648015905 conn(0x7fee400fb0b0 msgr2=0x7fee400fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:44.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.739+0000 7fee46c68700 1 -- 172.21.15.46:0/3648015905 shutdown_connections 2024-04-27T04:01:44.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.739+0000 7fee46c68700 1 -- 172.21.15.46:0/3648015905 wait complete. 2024-04-27T04:01:44.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.740+0000 7fee46c68700 1 Processor -- start 2024-04-27T04:01:44.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.740+0000 7fee46c68700 1 -- start start 2024-04-27T04:01:44.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.740+0000 7fee46c68700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee400ffb50 0x7fee40069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:44.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.740+0000 7fee46c68700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee40100d50 0x7fee4019b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:44.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.740+0000 7fee46c68700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4019b880 con 0x7fee400ffb50 2024-04-27T04:01:44.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.740+0000 7fee46c68700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4019b9f0 con 0x7fee40100d50 2024-04-27T04:01:44.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.740+0000 7fee3ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee40100d50 0x7fee4019b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:44.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.740+0000 7fee44a04700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee400ffb50 0x7fee40069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:44.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.740+0000 7fee3ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee40100d50 0x7fee4019b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.46:48770/0 (socket says 172.21.15.46:48770) 2024-04-27T04:01:44.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.740+0000 7fee44a04700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee400ffb50 0x7fee40069bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51100/0 (socket says 172.21.15.46:51100) 2024-04-27T04:01:44.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.740+0000 7fee3ffff700 1 -- 172.21.15.46:0/2058303343 learned_addr learned my addr 172.21.15.46:0/2058303343 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:44.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.740+0000 7fee3ffff700 1 -- 172.21.15.46:0/2058303343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee400ffb50 msgr2=0x7fee40069bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:44.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.740+0000 7fee3ffff700 1 --2- 172.21.15.46:0/2058303343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee400ffb50 0x7fee40069bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:44.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.740+0000 7fee3ffff700 1 -- 172.21.15.46:0/2058303343 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee34009520 con 0x7fee40100d50 2024-04-27T04:01:44.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.741+0000 7fee3ffff700 1 --2- 172.21.15.46:0/2058303343 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee40100d50 0x7fee4019b340 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00b800 tx=0x7fee2c005f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:44.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.741+0000 7fee3dffb700 1 -- 172.21.15.46:0/2058303343 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee2c01d070 con 0x7fee40100d50 2024-04-27T04:01:44.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.741+0000 7fee3dffb700 1 -- 172.21.15.46:0/2058303343 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee2c0052a0 con 0x7fee40100d50 2024-04-27T04:01:44.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.741+0000 7fee46c68700 1 -- 172.21.15.46:0/2058303343 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2c0095f0 con 0x7fee40100d50 2024-04-27T04:01:44.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.741+0000 7fee3dffb700 1 -- 172.21.15.46:0/2058303343 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee2c017400 con 0x7fee40100d50 2024-04-27T04:01:44.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.741+0000 7fee46c68700 1 -- 172.21.15.46:0/2058303343 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4019c050 con 0x7fee40100d50 2024-04-27T04:01:44.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.743+0000 7fee3dffb700 1 -- 172.21.15.46:0/2058303343 <== mon.1 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fee2c00f820 con 0x7fee40100d50 2024-04-27T04:01:44.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.743+0000 7fee46c68700 1 -- 172.21.15.46:0/2058303343 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4018f6d0 con 0x7fee40100d50 2024-04-27T04:01:44.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.743+0000 7fee3dffb700 1 --2- 172.21.15.46:0/2058303343 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fee3006cce0 0x7fee3006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:44.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.743+0000 7fee3dffb700 1 -- 172.21.15.46:0/2058303343 <== mon.1 v2:172.21.15.73:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fee2c08d6a0 con 0x7fee40100d50 2024-04-27T04:01:44.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.743+0000 7fee44a04700 1 --2- 172.21.15.46:0/2058303343 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fee3006cce0 0x7fee3006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:44.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.744+0000 7fee44a04700 1 --2- 172.21.15.46:0/2058303343 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fee3006cce0 0x7fee3006f1a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fee40100bb0 tx=0x7fee34009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:44.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:44.750+0000 7fee3dffb700 1 -- 172.21.15.46:0/2058303343 <== mon.1 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fee2c05b570 con 0x7fee40100d50 2024-04-27T04:01:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:45.032+0000 7fee46c68700 1 -- 172.21.15.46:0/2058303343 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee401088f0 con 0x7fee40100d50 2024-04-27T04:01:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:45.032+0000 7fee3dffb700 1 -- 172.21.15.46:0/2058303343 <== mon.1 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4853 (secure 0 0 0) 0x7fee2c05b100 con 0x7fee40100d50 2024-04-27T04:01:45.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:45.035+0000 7fee46c68700 1 -- 172.21.15.46:0/2058303343 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fee3006cce0 msgr2=0x7fee3006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:45.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:45.035+0000 7fee46c68700 1 --2- 172.21.15.46:0/2058303343 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fee3006cce0 0x7fee3006f1a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fee40100bb0 tx=0x7fee34009260 comp rx=0 tx=0).stop 2024-04-27T04:01:45.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:45.035+0000 7fee46c68700 1 -- 172.21.15.46:0/2058303343 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee40100d50 msgr2=0x7fee4019b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:45.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:45.035+0000 7fee46c68700 1 --2- 172.21.15.46:0/2058303343 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee40100d50 0x7fee4019b340 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00b800 tx=0x7fee2c005f00 comp rx=0 tx=0).stop 2024-04-27T04:01:45.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:45.035+0000 7fee46c68700 1 -- 172.21.15.46:0/2058303343 shutdown_connections 2024-04-27T04:01:45.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:45.035+0000 7fee46c68700 1 --2- 172.21.15.46:0/2058303343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee400ffb50 0x7fee40069bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:45.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:45.035+0000 7fee46c68700 1 --2- 172.21.15.46:0/2058303343 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fee3006cce0 0x7fee3006f1a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:45.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:45.035+0000 7fee46c68700 1 --2- 172.21.15.46:0/2058303343 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee40100d50 0x7fee4019b340 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:45.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:45.035+0000 7fee46c68700 1 -- 172.21.15.46:0/2058303343 >> 172.21.15.46:0/2058303343 conn(0x7fee400fb0b0 msgr2=0x7fee40104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:45.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:45.035+0000 7fee46c68700 1 -- 172.21.15.46:0/2058303343 shutdown_connections 2024-04-27T04:01:45.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:45.035+0000 7fee46c68700 1 -- 172.21.15.46:0/2058303343 wait complete. 2024-04-27T04:01:45.038 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 19 2024-04-27T04:01:45.047 INFO:teuthology.orchestra.run.smithi046.stdout:true 2024-04-27T04:01:45.290 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:45 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:45.290 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:45 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:45.290 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:45 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:01:45.290 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:45 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:01:45.290 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:45 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:45.290 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:45 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:01:45.290 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:45 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2058303343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T04:01:45.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:45 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:45.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:45 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:45.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:45 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:01:45.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:45 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:01:45.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:45 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:45.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:45 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:01:45.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:45 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/2058303343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T04:01:46.180 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-27T04:01:46.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:46 smithi073 ceph-mon[34160]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-27T04:01:46.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:46 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:46.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:46 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:01:46.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:46 smithi046 ceph-mon[26899]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-27T04:01:46.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:46 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:46.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:46 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:01:47.055 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:01:47.805 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:47 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:47.806 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:47 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:47.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:47 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:47.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:47 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:48.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:48 smithi046 ceph-mon[26899]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 10 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-27T04:01:49.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:48 smithi073 ceph-mon[34160]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 10 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-27T04:01:49.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.810+0000 7f2009ed4700 1 -- 172.21.15.46:0/1452815054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2004101e30 msgr2=0x7f2004102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:49.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.810+0000 7f2009ed4700 1 --2- 172.21.15.46:0/1452815054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2004101e30 0x7f2004102290 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8009910 tx=0x7f1ff8009c20 comp rx=0 tx=0).stop 2024-04-27T04:01:49.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.810+0000 7f2009ed4700 1 -- 172.21.15.46:0/1452815054 shutdown_connections 2024-04-27T04:01:49.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.810+0000 7f2009ed4700 1 --2- 172.21.15.46:0/1452815054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2004101e30 0x7f2004102290 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:49.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.810+0000 7f2009ed4700 1 --2- 172.21.15.46:0/1452815054 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2004100c30 0x7f2004101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:49.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.810+0000 7f2009ed4700 1 -- 172.21.15.46:0/1452815054 >> 172.21.15.46:0/1452815054 conn(0x7f20040fc1d0 msgr2=0x7f20040fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:49.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.810+0000 7f2009ed4700 1 -- 172.21.15.46:0/1452815054 shutdown_connections 2024-04-27T04:01:49.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.810+0000 7f2009ed4700 1 -- 172.21.15.46:0/1452815054 wait complete. 2024-04-27T04:01:49.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.811+0000 7f2009ed4700 1 Processor -- start 2024-04-27T04:01:49.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.811+0000 7f2009ed4700 1 -- start start 2024-04-27T04:01:49.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.811+0000 7f2009ed4700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2004100c30 0x7f2004196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:49.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.811+0000 7f2009ed4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2004101e30 0x7f2004196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:49.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.811+0000 7f2009ed4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20041971a0 con 0x7f2004101e30 2024-04-27T04:01:49.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.811+0000 7f2009ed4700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20041972e0 con 0x7f2004100c30 2024-04-27T04:01:49.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.811+0000 7f2002ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2004101e30 0x7f2004196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:49.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.811+0000 7f2002ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2004101e30 0x7f2004196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51122/0 (socket says 172.21.15.46:51122) 2024-04-27T04:01:49.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.811+0000 7f2002ffd700 1 -- 172.21.15.46:0/3030982849 learned_addr learned my addr 172.21.15.46:0/3030982849 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:49.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.812+0000 7f20037fe700 1 --2- 172.21.15.46:0/3030982849 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2004100c30 0x7f2004196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:49.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.812+0000 7f2002ffd700 1 -- 172.21.15.46:0/3030982849 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2004100c30 msgr2=0x7f2004196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:49.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.812+0000 7f2002ffd700 1 --2- 172.21.15.46:0/3030982849 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2004100c30 0x7f2004196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:49.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.812+0000 7f2002ffd700 1 -- 172.21.15.46:0/3030982849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff80095f0 con 0x7f2004101e30 2024-04-27T04:01:49.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.812+0000 7f20037fe700 1 --2- 172.21.15.46:0/3030982849 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2004100c30 0x7f2004196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T04:01:49.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.812+0000 7f2002ffd700 1 --2- 172.21.15.46:0/3030982849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2004101e30 0x7f2004196b80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80092e0 tx=0x7f1ff800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:49.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.812+0000 7f2000ff9700 1 -- 172.21.15.46:0/3030982849 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ff801d070 con 0x7f2004101e30 2024-04-27T04:01:49.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.812+0000 7f2000ff9700 1 -- 172.21.15.46:0/3030982849 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ff800f460 con 0x7f2004101e30 2024-04-27T04:01:49.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.812+0000 7f2009ed4700 1 -- 172.21.15.46:0/3030982849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f200419bc90 con 0x7f2004101e30 2024-04-27T04:01:49.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.812+0000 7f2000ff9700 1 -- 172.21.15.46:0/3030982849 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ff8021640 con 0x7f2004101e30 2024-04-27T04:01:49.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.812+0000 7f2009ed4700 1 -- 172.21.15.46:0/3030982849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f200419c200 con 0x7f2004101e30 2024-04-27T04:01:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.813+0000 7f2009ed4700 1 -- 172.21.15.46:0/3030982849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20041907b0 con 0x7f2004101e30 2024-04-27T04:01:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.814+0000 7f2000ff9700 1 -- 172.21.15.46:0/3030982849 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1ff800fad0 con 0x7f2004101e30 2024-04-27T04:01:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.814+0000 7f2000ff9700 1 --2- 172.21.15.46:0/3030982849 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1fec06cce0 0x7f1fec06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.814+0000 7f2000ff9700 1 -- 172.21.15.46:0/3030982849 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1ff808d500 con 0x7f2004101e30 2024-04-27T04:01:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.814+0000 7f20037fe700 1 --2- 172.21.15.46:0/3030982849 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1fec06cce0 0x7f1fec06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.815+0000 7f20037fe700 1 --2- 172.21.15.46:0/3030982849 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1fec06cce0 0x7f1fec06f1a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4005950 tx=0x7f1ff40058e0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:49.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:49.819+0000 7f2000ff9700 1 -- 172.21.15.46:0/3030982849 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f1ff805b300 con 0x7f2004101e30 2024-04-27T04:01:50.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:50.112+0000 7f2009ed4700 1 -- 172.21.15.46:0/3030982849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f200404ebb0 con 0x7f2004101e30 2024-04-27T04:01:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:50.112+0000 7f2000ff9700 1 -- 172.21.15.46:0/3030982849 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v19) v1 ==== 78+0+98 (secure 0 0 0) 0x7f1ff805ae90 con 0x7f2004101e30 2024-04-27T04:01:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:50.114+0000 7f2009ed4700 1 -- 172.21.15.46:0/3030982849 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1fec06cce0 msgr2=0x7f1fec06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:50.114+0000 7f2009ed4700 1 --2- 172.21.15.46:0/3030982849 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1fec06cce0 0x7f1fec06f1a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4005950 tx=0x7f1ff40058e0 comp rx=0 tx=0).stop 2024-04-27T04:01:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:50.114+0000 7f2009ed4700 1 -- 172.21.15.46:0/3030982849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2004101e30 msgr2=0x7f2004196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:50.114+0000 7f2009ed4700 1 --2- 172.21.15.46:0/3030982849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2004101e30 0x7f2004196b80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80092e0 tx=0x7f1ff800be20 comp rx=0 tx=0).stop 2024-04-27T04:01:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:50.114+0000 7f2009ed4700 1 -- 172.21.15.46:0/3030982849 shutdown_connections 2024-04-27T04:01:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:50.114+0000 7f2009ed4700 1 --2- 172.21.15.46:0/3030982849 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1fec06cce0 0x7f1fec06f1a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:50.114+0000 7f2009ed4700 1 --2- 172.21.15.46:0/3030982849 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2004100c30 0x7f2004196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:50.114+0000 7f2009ed4700 1 --2- 172.21.15.46:0/3030982849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2004101e30 0x7f2004196b80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:50.114+0000 7f2009ed4700 1 -- 172.21.15.46:0/3030982849 >> 172.21.15.46:0/3030982849 conn(0x7f20040fc1d0 msgr2=0x7f20041053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:50.114+0000 7f2009ed4700 1 -- 172.21.15.46:0/3030982849 shutdown_connections 2024-04-27T04:01:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:50.114+0000 7f2009ed4700 1 -- 172.21.15.46:0/3030982849 wait complete. 2024-04-27T04:01:50.126 INFO:teuthology.orchestra.run.smithi046.stdout:true 2024-04-27T04:01:50.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:50 smithi046 ceph-mon[26899]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-27T04:01:50.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:50 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/3030982849' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-27T04:01:51.052 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-27T04:01:51.065 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T04:01:51.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:50 smithi073 ceph-mon[34160]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-27T04:01:51.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:50 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/3030982849' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-27T04:01:51.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.200+0000 7fa98088f640 1 -- 172.21.15.46:0/886655203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa97c0fe9d0 msgr2=0x7fa97c0fedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:51.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.200+0000 7fa98088f640 1 --2- 172.21.15.46:0/886655203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa97c0fe9d0 0x7fa97c0fedf0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa96c009880 tx=0x7fa96c02bce0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.200+0000 7fa98088f640 1 -- 172.21.15.46:0/886655203 shutdown_connections 2024-04-27T04:01:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.200+0000 7fa98088f640 1 --2- 172.21.15.46:0/886655203 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa97c0ff330 0x7fa97c0ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.200+0000 7fa98088f640 1 --2- 172.21.15.46:0/886655203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa97c0fe9d0 0x7fa97c0fedf0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.200+0000 7fa98088f640 1 -- 172.21.15.46:0/886655203 >> 172.21.15.46:0/886655203 conn(0x7fa97c0fa5d0 msgr2=0x7fa97c0fc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.200+0000 7fa98088f640 1 -- 172.21.15.46:0/886655203 shutdown_connections 2024-04-27T04:01:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.200+0000 7fa98088f640 1 -- 172.21.15.46:0/886655203 wait complete. 2024-04-27T04:01:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.201+0000 7fa98088f640 1 Processor -- start 2024-04-27T04:01:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.201+0000 7fa98088f640 1 -- start start 2024-04-27T04:01:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.201+0000 7fa98088f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa97c0fe9d0 0x7fa97c197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.201+0000 7fa98088f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa97c0ff330 0x7fa97c198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.201+0000 7fa98088f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c198960 con 0x7fa97c0fe9d0 2024-04-27T04:01:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.201+0000 7fa98088f640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c198ad0 con 0x7fa97c0ff330 2024-04-27T04:01:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.201+0000 7fa97a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa97c0fe9d0 0x7fa97c197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.201+0000 7fa97a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa97c0fe9d0 0x7fa97c197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38796/0 (socket says 172.21.15.46:38796) 2024-04-27T04:01:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.201+0000 7fa97a59c640 1 -- 172.21.15.46:0/9959520 learned_addr learned my addr 172.21.15.46:0/9959520 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.201+0000 7fa973fff640 1 --2- 172.21.15.46:0/9959520 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa97c0ff330 0x7fa97c198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.201+0000 7fa97a59c640 1 -- 172.21.15.46:0/9959520 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa97c0ff330 msgr2=0x7fa97c198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.201+0000 7fa97a59c640 1 --2- 172.21.15.46:0/9959520 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa97c0ff330 0x7fa97c198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.201+0000 7fa97a59c640 1 -- 172.21.15.46:0/9959520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96c009560 con 0x7fa97c0fe9d0 2024-04-27T04:01:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.201+0000 7fa973fff640 1 --2- 172.21.15.46:0/9959520 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa97c0ff330 0x7fa97c198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T04:01:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.202+0000 7fa97a59c640 1 --2- 172.21.15.46:0/9959520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa97c0fe9d0 0x7fa97c197e50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa96c036040 tx=0x7fa96c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.202+0000 7fa9737fe640 1 -- 172.21.15.46:0/9959520 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa96c004520 con 0x7fa97c0fe9d0 2024-04-27T04:01:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.202+0000 7fa9737fe640 1 -- 172.21.15.46:0/9959520 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa96c003ab0 con 0x7fa97c0fe9d0 2024-04-27T04:01:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.202+0000 7fa98088f640 1 -- 172.21.15.46:0/9959520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa97c19d4b0 con 0x7fa97c0fe9d0 2024-04-27T04:01:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.202+0000 7fa9737fe640 1 -- 172.21.15.46:0/9959520 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa96c005670 con 0x7fa97c0fe9d0 2024-04-27T04:01:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.202+0000 7fa98088f640 1 -- 172.21.15.46:0/9959520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa97c19da30 con 0x7fa97c0fe9d0 2024-04-27T04:01:51.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.203+0000 7fa98088f640 1 -- 172.21.15.46:0/9959520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa97c191e80 con 0x7fa97c0fe9d0 2024-04-27T04:01:51.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.203+0000 7fa9737fe640 1 -- 172.21.15.46:0/9959520 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa96c0035f0 con 0x7fa97c0fe9d0 2024-04-27T04:01:51.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.204+0000 7fa9737fe640 1 --2- 172.21.15.46:0/9959520 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fa96806cbf0 0x7fa96806f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:51.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.204+0000 7fa9737fe640 1 -- 172.21.15.46:0/9959520 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa96c0ae5d0 con 0x7fa97c0fe9d0 2024-04-27T04:01:51.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.204+0000 7fa973fff640 1 --2- 172.21.15.46:0/9959520 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fa96806cbf0 0x7fa96806f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:51.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.204+0000 7fa973fff640 1 --2- 172.21.15.46:0/9959520 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fa96806cbf0 0x7fa96806f0b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa97c199310 tx=0x7fa964005f90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:51.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.209+0000 7fa9737fe640 1 -- 172.21.15.46:0/9959520 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fa96c0789f0 con 0x7fa97c0fe9d0 2024-04-27T04:01:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.395+0000 7fa98088f640 1 -- 172.21.15.46:0/9959520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa97c02c3b0 con 0x7fa97c0fe9d0 2024-04-27T04:01:51.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.396+0000 7fa9737fe640 1 -- 172.21.15.46:0/9959520 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+15104 (secure 0 0 0) 0x7fa96c07c010 con 0x7fa97c0fe9d0 2024-04-27T04:01:51.397 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:01:51.398 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":46,"fsid":"b1d3a2d0-0449-11ef-bc93-c7b262605968","created":"2024-04-27T03:54:08.502636+0000","modified":"2024-04-27T04:01:38.107789+0000","last_up_change":"2024-04-27T04:00:02.972167+0000","last_in_change":"2024-04-27T03:59:43.096341+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T03:58:45.241461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-27T04:01:20.273215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-27T04:01:21.104548+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0310053-0895-4608-9021-6c04075a1f9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2222489853},{"type":"v1","addr":"172.21.15.46:6803","nonce":2222489853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2222489853},{"type":"v1","addr":"172.21.15.46:6805","nonce":2222489853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2222489853},{"type":"v1","addr":"172.21.15.46:6809","nonce":2222489853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2222489853},{"type":"v1","addr":"172.21.15.46:6807","nonce":2222489853}]},"public_addr":"172.21.15.46:6803/2222489853","cluster_addr":"172.21.15.46:6805/2222489853","heartbeat_back_addr":"172.21.15.46:6809/2222489853","heartbeat_front_addr":"172.21.15.46:6807/2222489853","state":["exists","up"]},{"osd":1,"uuid":"57d1cc25-d5db-41fd-aa72-e380cedf2225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":1807987418},{"type":"v1","addr":"172.21.15.46:6811","nonce":1807987418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":1807987418},{"type":"v1","addr":"172.21.15.46:6813","nonce":1807987418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":1807987418},{"type":"v1","addr":"172.21.15.46:6817","nonce":1807987418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":1807987418},{"type":"v1","addr":"172.21.15.46:6815","nonce":1807987418}]},"public_addr":"172.21.15.46:6811/1807987418","cluster_addr":"172.21.15.46:6813/1807987418","heartbeat_back_addr":"172.21.15.46:6817/1807987418","heartbeat_front_addr":"172.21.15.46:6815/1807987418","state":["exists","up"]},{"osd":2,"uuid":"addf9aa9-ba34-4443-b239-6df9f7ef68e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":933743760},{"type":"v1","addr":"172.21.15.46:6819","nonce":933743760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":933743760},{"type":"v1","addr":"172.21.15.46:6821","nonce":933743760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":933743760},{"type":"v1","addr":"172.21.15.46:6825","nonce":933743760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":933743760},{"type":"v1","addr":"172.21.15.46:6823","nonce":933743760}]},"public_addr":"172.21.15.46:6819/933743760","cluster_addr":"172.21.15.46:6821/933743760","heartbeat_back_addr":"172.21.15.46:6825/933743760","heartbeat_front_addr":"172.21.15.46:6823/933743760","state":["exists","up"]},{"osd":3,"uuid":"4fa6eb8e-8816-4308-894d-575c57e335d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":2427344041},{"type":"v1","addr":"172.21.15.73:6801","nonce":2427344041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":2427344041},{"type":"v1","addr":"172.21.15.73:6803","nonce":2427344041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":2427344041},{"type":"v1","addr":"172.21.15.73:6807","nonce":2427344041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":2427344041},{"type":"v1","addr":"172.21.15.73:6805","nonce":2427344041}]},"public_addr":"172.21.15.73:6801/2427344041","cluster_addr":"172.21.15.73:6803/2427344041","heartbeat_back_addr":"172.21.15.73:6807/2427344041","heartbeat_front_addr":"172.21.15.73:6805/2427344041","state":["exists","up"]},{"osd":4,"uuid":"fb0a02c1-cc1a-4c2a-a469-0c941b9fe3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":1194795747},{"type":"v1","addr":"172.21.15.73:6809","nonce":1194795747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":1194795747},{"type":"v1","addr":"172.21.15.73:6811","nonce":1194795747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":1194795747},{"type":"v1","addr":"172.21.15.73:6815","nonce":1194795747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":1194795747},{"type":"v1","addr":"172.21.15.73:6813","nonce":1194795747}]},"public_addr":"172.21.15.73:6809/1194795747","cluster_addr":"172.21.15.73:6811/1194795747","heartbeat_back_addr":"172.21.15.73:6815/1194795747","heartbeat_front_addr":"172.21.15.73:6813/1194795747","state":["exists","up"]},{"osd":5,"uuid":"b9d4c4d4-5633-4e9a-ba64-460134d132ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":4277072929},{"type":"v1","addr":"172.21.15.73:6817","nonce":4277072929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":4277072929},{"type":"v1","addr":"172.21.15.73:6819","nonce":4277072929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":4277072929},{"type":"v1","addr":"172.21.15.73:6823","nonce":4277072929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":4277072929},{"type":"v1","addr":"172.21.15.73:6821","nonce":4277072929}]},"public_addr":"172.21.15.73:6817/4277072929","cluster_addr":"172.21.15.73:6819/4277072929","heartbeat_back_addr":"172.21.15.73:6823/4277072929","heartbeat_front_addr":"172.21.15.73:6821/4277072929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:57:47.645830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:58:15.796052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:58:43.469886+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:59:08.833298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:59:34.410852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T04:00:00.927552+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6827/2151398550":"2024-04-28T04:01:38.107765+0000","172.21.15.46:6801/3824329935":"2024-04-28T03:54:40.659687+0000","172.21.15.46:6800/333567692":"2024-04-28T03:55:14.071598+0000","172.21.15.46:0/1611290489":"2024-04-28T03:56:30.716510+0000","172.21.15.46:0/3576221786":"2024-04-28T03:55:14.071598+0000","172.21.15.46:0/1243800260":"2024-04-28T03:55:14.071598+0000","172.21.15.46:0/3828444952":"2024-04-28T03:54:40.659687+0000","172.21.15.46:6800/3824329935":"2024-04-28T03:54:40.659687+0000","172.21.15.46:6801/333567692":"2024-04-28T03:55:14.071598+0000","172.21.15.46:0/2660836044":"2024-04-28T03:54:40.659687+0000","172.21.15.46:6826/2151398550":"2024-04-28T04:01:38.107765+0000","172.21.15.46:0/1318513098":"2024-04-28T03:54:40.659687+0000","172.21.15.46:0/2377183342":"2024-04-28T03:55:14.071598+0000","172.21.15.46:6801/3280787672":"2024-04-28T03:56:30.716510+0000","172.21.15.46:0/1728399059":"2024-04-28T03:56:30.716510+0000","172.21.15.46:0/3592746122":"2024-04-28T03:56:30.716510+0000","172.21.15.46:6800/3280787672":"2024-04-28T03:56:30.716510+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T04:01:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.399+0000 7fa98088f640 1 -- 172.21.15.46:0/9959520 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fa96806cbf0 msgr2=0x7fa96806f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.399+0000 7fa98088f640 1 --2- 172.21.15.46:0/9959520 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fa96806cbf0 0x7fa96806f0b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa97c199310 tx=0x7fa964005f90 comp rx=0 tx=0).stop 2024-04-27T04:01:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.399+0000 7fa98088f640 1 -- 172.21.15.46:0/9959520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa97c0fe9d0 msgr2=0x7fa97c197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.399+0000 7fa98088f640 1 --2- 172.21.15.46:0/9959520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa97c0fe9d0 0x7fa97c197e50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa96c036040 tx=0x7fa96c005dc0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.399+0000 7fa98088f640 1 -- 172.21.15.46:0/9959520 shutdown_connections 2024-04-27T04:01:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.399+0000 7fa98088f640 1 --2- 172.21.15.46:0/9959520 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fa96806cbf0 0x7fa96806f0b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.399+0000 7fa98088f640 1 --2- 172.21.15.46:0/9959520 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa97c0ff330 0x7fa97c198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.399+0000 7fa98088f640 1 --2- 172.21.15.46:0/9959520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa97c0fe9d0 0x7fa97c197e50 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.399+0000 7fa98088f640 1 -- 172.21.15.46:0/9959520 >> 172.21.15.46:0/9959520 conn(0x7fa97c0fa5d0 msgr2=0x7fa97c107420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.399+0000 7fa98088f640 1 -- 172.21.15.46:0/9959520 shutdown_connections 2024-04-27T04:01:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.399+0000 7fa98088f640 1 -- 172.21.15.46:0/9959520 wait complete. 2024-04-27T04:01:51.411 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-27T03:58:45.241461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-27T04:01:20.273215+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-27T04:01:21.104548+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-27T04:01:51.412 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-27T04:01:51.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.514+0000 7f9e1e9eb640 1 -- 172.21.15.46:0/3183274901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e18100b80 msgr2=0x7f9e18102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:51.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.514+0000 7f9e1e9eb640 1 --2- 172.21.15.46:0/3183274901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e18100b80 0x7f9e18102f70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9e080098b0 tx=0x7f9e0802bcd0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.514+0000 7f9e1e9eb640 1 -- 172.21.15.46:0/3183274901 shutdown_connections 2024-04-27T04:01:51.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.514+0000 7f9e1e9eb640 1 --2- 172.21.15.46:0/3183274901 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e181034b0 0x7f9e181058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.514+0000 7f9e1e9eb640 1 --2- 172.21.15.46:0/3183274901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e18100b80 0x7f9e18102f70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.514+0000 7f9e1e9eb640 1 -- 172.21.15.46:0/3183274901 >> 172.21.15.46:0/3183274901 conn(0x7f9e180fa5d0 msgr2=0x7f9e180fc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:51.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e1e9eb640 1 -- 172.21.15.46:0/3183274901 shutdown_connections 2024-04-27T04:01:51.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e1e9eb640 1 -- 172.21.15.46:0/3183274901 wait complete. 2024-04-27T04:01:51.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e1e9eb640 1 Processor -- start 2024-04-27T04:01:51.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e1e9eb640 1 -- start start 2024-04-27T04:01:51.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e1e9eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e18100b80 0x7f9e18195c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:51.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e1e9eb640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e181034b0 0x7f9e18196170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:51.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e1e9eb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e181966b0 con 0x7f9e18100b80 2024-04-27T04:01:51.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e1e9eb640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e18196820 con 0x7f9e181034b0 2024-04-27T04:01:51.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e17fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e18100b80 0x7f9e18195c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:51.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e17fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e18100b80 0x7f9e18195c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38810/0 (socket says 172.21.15.46:38810) 2024-04-27T04:01:51.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e17fff640 1 -- 172.21.15.46:0/2013819789 learned_addr learned my addr 172.21.15.46:0/2013819789 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:51.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e177fe640 1 --2- 172.21.15.46:0/2013819789 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e181034b0 0x7f9e18196170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:51.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e17fff640 1 -- 172.21.15.46:0/2013819789 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e181034b0 msgr2=0x7f9e18196170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:51.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e17fff640 1 --2- 172.21.15.46:0/2013819789 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e181034b0 0x7f9e18196170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e17fff640 1 -- 172.21.15.46:0/2013819789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e08009560 con 0x7f9e18100b80 2024-04-27T04:01:51.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e177fe640 1 --2- 172.21.15.46:0/2013819789 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e181034b0 0x7f9e18196170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T04:01:51.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e17fff640 1 --2- 172.21.15.46:0/2013819789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e18100b80 0x7f9e18195c30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9e08009430 tx=0x7f9e080042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:51.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e157fa640 1 -- 172.21.15.46:0/2013819789 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e080039d0 con 0x7f9e18100b80 2024-04-27T04:01:51.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e157fa640 1 -- 172.21.15.46:0/2013819789 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e080044e0 con 0x7f9e18100b80 2024-04-27T04:01:51.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e1e9eb640 1 -- 172.21.15.46:0/2013819789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1819b240 con 0x7f9e18100b80 2024-04-27T04:01:51.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e157fa640 1 -- 172.21.15.46:0/2013819789 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e08034440 con 0x7f9e18100b80 2024-04-27T04:01:51.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.516+0000 7f9e1e9eb640 1 -- 172.21.15.46:0/2013819789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1819b790 con 0x7f9e18100b80 2024-04-27T04:01:51.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.517+0000 7f9e1e9eb640 1 -- 172.21.15.46:0/2013819789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1804e160 con 0x7f9e18100b80 2024-04-27T04:01:51.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.518+0000 7f9e157fa640 1 -- 172.21.15.46:0/2013819789 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9e08004ff0 con 0x7f9e18100b80 2024-04-27T04:01:51.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.518+0000 7f9e157fa640 1 --2- 172.21.15.46:0/2013819789 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9e0006cc60 0x7f9e0006f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:51.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.518+0000 7f9e177fe640 1 --2- 172.21.15.46:0/2013819789 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9e0006cc60 0x7f9e0006f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:51.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.518+0000 7f9e157fa640 1 -- 172.21.15.46:0/2013819789 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f9e080ae840 con 0x7f9e18100b80 2024-04-27T04:01:51.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.520+0000 7f9e177fe640 1 --2- 172.21.15.46:0/2013819789 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9e0006cc60 0x7f9e0006f120 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9e181970f0 tx=0x7f9e0c00a5c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:51.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.524+0000 7f9e157fa640 1 -- 172.21.15.46:0/2013819789 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f9e08078bb0 con 0x7f9e18100b80 2024-04-27T04:01:51.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.706+0000 7f9e1e9eb640 1 -- 172.21.15.46:0/2013819789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9e1819b9e0 con 0x7f9e18100b80 2024-04-27T04:01:51.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.706+0000 7f9e157fa640 1 -- 172.21.15.46:0/2013819789 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9e0807c1d0 con 0x7f9e18100b80 2024-04-27T04:01:51.707 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-27T04:01:51.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.708+0000 7f9e1e9eb640 1 -- 172.21.15.46:0/2013819789 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9e0006cc60 msgr2=0x7f9e0006f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:51.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.708+0000 7f9e1e9eb640 1 --2- 172.21.15.46:0/2013819789 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9e0006cc60 0x7f9e0006f120 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9e181970f0 tx=0x7f9e0c00a5c0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.708+0000 7f9e1e9eb640 1 -- 172.21.15.46:0/2013819789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e18100b80 msgr2=0x7f9e18195c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:51.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.708+0000 7f9e1e9eb640 1 --2- 172.21.15.46:0/2013819789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e18100b80 0x7f9e18195c30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9e08009430 tx=0x7f9e080042f0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.708+0000 7f9e1e9eb640 1 -- 172.21.15.46:0/2013819789 shutdown_connections 2024-04-27T04:01:51.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.708+0000 7f9e1e9eb640 1 --2- 172.21.15.46:0/2013819789 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9e0006cc60 0x7f9e0006f120 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.708+0000 7f9e1e9eb640 1 --2- 172.21.15.46:0/2013819789 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e181034b0 0x7f9e18196170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.708+0000 7f9e1e9eb640 1 --2- 172.21.15.46:0/2013819789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e18100b80 0x7f9e18195c30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.708+0000 7f9e1e9eb640 1 -- 172.21.15.46:0/2013819789 >> 172.21.15.46:0/2013819789 conn(0x7f9e180fa5d0 msgr2=0x7f9e180fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:51.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.708+0000 7f9e1e9eb640 1 -- 172.21.15.46:0/2013819789 shutdown_connections 2024-04-27T04:01:51.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.708+0000 7f9e1e9eb640 1 -- 172.21.15.46:0/2013819789 wait complete. 2024-04-27T04:01:51.719 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-27T04:01:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.823+0000 7f2eadcaf640 1 -- 172.21.15.46:0/3856374662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ea8100be0 msgr2=0x7f2ea8102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.823+0000 7f2eadcaf640 1 --2- 172.21.15.46:0/3856374662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ea8100be0 0x7f2ea8102fd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009880 tx=0x7f2e9002bce0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.823+0000 7f2eadcaf640 1 -- 172.21.15.46:0/3856374662 shutdown_connections 2024-04-27T04:01:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.823+0000 7f2eadcaf640 1 --2- 172.21.15.46:0/3856374662 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ea8103510 0x7f2ea8105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.823+0000 7f2eadcaf640 1 --2- 172.21.15.46:0/3856374662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ea8100be0 0x7f2ea8102fd0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.823+0000 7f2eadcaf640 1 -- 172.21.15.46:0/3856374662 >> 172.21.15.46:0/3856374662 conn(0x7f2ea80fa550 msgr2=0x7f2ea80fc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.823+0000 7f2eadcaf640 1 -- 172.21.15.46:0/3856374662 shutdown_connections 2024-04-27T04:01:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.823+0000 7f2eadcaf640 1 -- 172.21.15.46:0/3856374662 wait complete. 2024-04-27T04:01:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.824+0000 7f2eadcaf640 1 Processor -- start 2024-04-27T04:01:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.824+0000 7f2eadcaf640 1 -- start start 2024-04-27T04:01:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.824+0000 7f2eadcaf640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ea8100be0 0x7f2ea8195c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.824+0000 7f2eadcaf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ea8103510 0x7f2ea81961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.824+0000 7f2eadcaf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea8196790 con 0x7f2ea8103510 2024-04-27T04:01:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.824+0000 7f2eadcaf640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea8196900 con 0x7f2ea8100be0 2024-04-27T04:01:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.824+0000 7f2ea6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ea8103510 0x7f2ea81961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.824+0000 7f2ea6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ea8103510 0x7f2ea81961c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38824/0 (socket says 172.21.15.46:38824) 2024-04-27T04:01:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.824+0000 7f2ea6ffd640 1 -- 172.21.15.46:0/4167041233 learned_addr learned my addr 172.21.15.46:0/4167041233 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.825+0000 7f2ea6ffd640 1 -- 172.21.15.46:0/4167041233 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ea8100be0 msgr2=0x7f2ea8195c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.825+0000 7f2ea77fe640 1 --2- 172.21.15.46:0/4167041233 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ea8100be0 0x7f2ea8195c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.825+0000 7f2ea6ffd640 1 --2- 172.21.15.46:0/4167041233 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ea8100be0 0x7f2ea8195c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.825+0000 7f2ea6ffd640 1 -- 172.21.15.46:0/4167041233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e90009560 con 0x7f2ea8103510 2024-04-27T04:01:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.825+0000 7f2ea6ffd640 1 --2- 172.21.15.46:0/4167041233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ea8103510 0x7f2ea81961c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2e9800d6c0 tx=0x7f2e9800db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.825+0000 7f2ea4ff9640 1 -- 172.21.15.46:0/4167041233 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e980051f0 con 0x7f2ea8103510 2024-04-27T04:01:51.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.825+0000 7f2eadcaf640 1 -- 172.21.15.46:0/4167041233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea819b330 con 0x7f2ea8103510 2024-04-27T04:01:51.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.825+0000 7f2ea4ff9640 1 -- 172.21.15.46:0/4167041233 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e98009ce0 con 0x7f2ea8103510 2024-04-27T04:01:51.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.825+0000 7f2ea4ff9640 1 -- 172.21.15.46:0/4167041233 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e98010460 con 0x7f2ea8103510 2024-04-27T04:01:51.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.825+0000 7f2eadcaf640 1 -- 172.21.15.46:0/4167041233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea819b910 con 0x7f2ea8103510 2024-04-27T04:01:51.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.826+0000 7f2eadcaf640 1 -- 172.21.15.46:0/4167041233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea8108cf0 con 0x7f2ea8103510 2024-04-27T04:01:51.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.827+0000 7f2ea4ff9640 1 -- 172.21.15.46:0/4167041233 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2e98005350 con 0x7f2ea8103510 2024-04-27T04:01:51.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.827+0000 7f2ea4ff9640 1 --2- 172.21.15.46:0/4167041233 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2e9406cc50 0x7f2e9406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:51.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.827+0000 7f2ea4ff9640 1 -- 172.21.15.46:0/4167041233 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2e9808bda0 con 0x7f2ea8103510 2024-04-27T04:01:51.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.827+0000 7f2ea77fe640 1 --2- 172.21.15.46:0/4167041233 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2e9406cc50 0x7f2e9406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:51.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.827+0000 7f2ea77fe640 1 --2- 172.21.15.46:0/4167041233 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2e9406cc50 0x7f2e9406f110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2e90036040 tx=0x7f2e90000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:51.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:51.832+0000 7f2ea4ff9640 1 -- 172.21.15.46:0/4167041233 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f2e98056110 con 0x7f2ea8103510 2024-04-27T04:01:52.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.011+0000 7f2eadcaf640 1 -- 172.21.15.46:0/4167041233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f2ea819bb30 con 0x7f2ea8103510 2024-04-27T04:01:52.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.012+0000 7f2ea4ff9640 1 -- 172.21.15.46:0/4167041233 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2e98059730 con 0x7f2ea8103510 2024-04-27T04:01:52.013 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-27T04:01:52.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.014+0000 7f2eadcaf640 1 -- 172.21.15.46:0/4167041233 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2e9406cc50 msgr2=0x7f2e9406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:52.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.014+0000 7f2eadcaf640 1 --2- 172.21.15.46:0/4167041233 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2e9406cc50 0x7f2e9406f110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2e90036040 tx=0x7f2e90000bc0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.014+0000 7f2eadcaf640 1 -- 172.21.15.46:0/4167041233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ea8103510 msgr2=0x7f2ea81961c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:52.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.014+0000 7f2eadcaf640 1 --2- 172.21.15.46:0/4167041233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ea8103510 0x7f2ea81961c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2e9800d6c0 tx=0x7f2e9800db80 comp rx=0 tx=0).stop 2024-04-27T04:01:52.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.014+0000 7f2eadcaf640 1 -- 172.21.15.46:0/4167041233 shutdown_connections 2024-04-27T04:01:52.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.014+0000 7f2eadcaf640 1 --2- 172.21.15.46:0/4167041233 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f2e9406cc50 0x7f2e9406f110 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.014+0000 7f2eadcaf640 1 --2- 172.21.15.46:0/4167041233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ea8103510 0x7f2ea81961c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.014+0000 7f2eadcaf640 1 --2- 172.21.15.46:0/4167041233 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ea8100be0 0x7f2ea8195c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.014+0000 7f2eadcaf640 1 -- 172.21.15.46:0/4167041233 >> 172.21.15.46:0/4167041233 conn(0x7f2ea80fa550 msgr2=0x7f2ea80fc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:52.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.014+0000 7f2eadcaf640 1 -- 172.21.15.46:0/4167041233 shutdown_connections 2024-04-27T04:01:52.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.014+0000 7f2eadcaf640 1 -- 172.21.15.46:0/4167041233 wait complete. 2024-04-27T04:01:52.026 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-27T04:01:52.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.129+0000 7fca28bd0640 1 -- 172.21.15.46:0/1697571637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca24100c00 msgr2=0x7fca24102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.129+0000 7fca28bd0640 1 --2- 172.21.15.46:0/1697571637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca24100c00 0x7fca24102ff0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fca1c009880 tx=0x7fca1c02bce0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.130+0000 7fca28bd0640 1 -- 172.21.15.46:0/1697571637 shutdown_connections 2024-04-27T04:01:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.130+0000 7fca28bd0640 1 --2- 172.21.15.46:0/1697571637 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fca24103530 0x7fca24105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.130+0000 7fca28bd0640 1 --2- 172.21.15.46:0/1697571637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca24100c00 0x7fca24102ff0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.130+0000 7fca28bd0640 1 -- 172.21.15.46:0/1697571637 >> 172.21.15.46:0/1697571637 conn(0x7fca240fa550 msgr2=0x7fca240fc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.130+0000 7fca28bd0640 1 -- 172.21.15.46:0/1697571637 shutdown_connections 2024-04-27T04:01:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.130+0000 7fca28bd0640 1 -- 172.21.15.46:0/1697571637 wait complete. 2024-04-27T04:01:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.130+0000 7fca28bd0640 1 Processor -- start 2024-04-27T04:01:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.130+0000 7fca28bd0640 1 -- start start 2024-04-27T04:01:52.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.131+0000 7fca28bd0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca24100c00 0x7fca24197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:52.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.131+0000 7fca28bd0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fca24103530 0x7fca24198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:52.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.131+0000 7fca28bd0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca24198960 con 0x7fca24100c00 2024-04-27T04:01:52.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.131+0000 7fca28bd0640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca24198ad0 con 0x7fca24103530 2024-04-27T04:01:52.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.131+0000 7fca2259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca24100c00 0x7fca24197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:52.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.131+0000 7fca2259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca24100c00 0x7fca24197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38846/0 (socket says 172.21.15.46:38846) 2024-04-27T04:01:52.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.131+0000 7fca2259c640 1 -- 172.21.15.46:0/55987853 learned_addr learned my addr 172.21.15.46:0/55987853 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:52.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.131+0000 7fca21d9b640 1 --2- 172.21.15.46:0/55987853 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fca24103530 0x7fca24198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:52.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.131+0000 7fca2259c640 1 -- 172.21.15.46:0/55987853 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fca24103530 msgr2=0x7fca24198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:52.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.131+0000 7fca2259c640 1 --2- 172.21.15.46:0/55987853 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fca24103530 0x7fca24198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.131+0000 7fca2259c640 1 -- 172.21.15.46:0/55987853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1c009560 con 0x7fca24100c00 2024-04-27T04:01:52.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.131+0000 7fca2259c640 1 --2- 172.21.15.46:0/55987853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca24100c00 0x7fca24197e50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fca1c036040 tx=0x7fca1c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:52.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.131+0000 7fca1b7fe640 1 -- 172.21.15.46:0/55987853 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca1c0035f0 con 0x7fca24100c00 2024-04-27T04:01:52.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.131+0000 7fca1b7fe640 1 -- 172.21.15.46:0/55987853 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca1c003c10 con 0x7fca24100c00 2024-04-27T04:01:52.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.131+0000 7fca1b7fe640 1 -- 172.21.15.46:0/55987853 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca1c034440 con 0x7fca24100c00 2024-04-27T04:01:52.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.131+0000 7fca28bd0640 1 -- 172.21.15.46:0/55987853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2419d4b0 con 0x7fca24100c00 2024-04-27T04:01:52.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.132+0000 7fca28bd0640 1 -- 172.21.15.46:0/55987853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2419d9b0 con 0x7fca24100c00 2024-04-27T04:01:52.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.132+0000 7fca28bd0640 1 -- 172.21.15.46:0/55987853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca24108cd0 con 0x7fca24100c00 2024-04-27T04:01:52.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.133+0000 7fca1b7fe640 1 -- 172.21.15.46:0/55987853 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fca1c004ff0 con 0x7fca24100c00 2024-04-27T04:01:52.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.133+0000 7fca1b7fe640 1 --2- 172.21.15.46:0/55987853 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fca1006cc50 0x7fca1006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:52.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.133+0000 7fca1b7fe640 1 -- 172.21.15.46:0/55987853 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fca1c0ae650 con 0x7fca24100c00 2024-04-27T04:01:52.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.133+0000 7fca21d9b640 1 --2- 172.21.15.46:0/55987853 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fca1006cc50 0x7fca1006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:52.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.134+0000 7fca21d9b640 1 --2- 172.21.15.46:0/55987853 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fca1006cc50 0x7fca1006f110 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fca24199310 tx=0x7fca0c00b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:52.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.138+0000 7fca1b7fe640 1 -- 172.21.15.46:0/55987853 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fca1c078a70 con 0x7fca24100c00 2024-04-27T04:01:52.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.318+0000 7fca28bd0640 1 -- 172.21.15.46:0/55987853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fca24108e60 con 0x7fca24100c00 2024-04-27T04:01:52.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.319+0000 7fca1b7fe640 1 -- 172.21.15.46:0/55987853 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7fca1c07c090 con 0x7fca24100c00 2024-04-27T04:01:52.319 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-27T04:01:52.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.321+0000 7fca28bd0640 1 -- 172.21.15.46:0/55987853 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fca1006cc50 msgr2=0x7fca1006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:52.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.321+0000 7fca28bd0640 1 --2- 172.21.15.46:0/55987853 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fca1006cc50 0x7fca1006f110 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fca24199310 tx=0x7fca0c00b410 comp rx=0 tx=0).stop 2024-04-27T04:01:52.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.321+0000 7fca28bd0640 1 -- 172.21.15.46:0/55987853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca24100c00 msgr2=0x7fca24197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:52.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.321+0000 7fca28bd0640 1 --2- 172.21.15.46:0/55987853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca24100c00 0x7fca24197e50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fca1c036040 tx=0x7fca1c004040 comp rx=0 tx=0).stop 2024-04-27T04:01:52.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.321+0000 7fca28bd0640 1 -- 172.21.15.46:0/55987853 shutdown_connections 2024-04-27T04:01:52.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.321+0000 7fca28bd0640 1 --2- 172.21.15.46:0/55987853 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fca1006cc50 0x7fca1006f110 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.321+0000 7fca28bd0640 1 --2- 172.21.15.46:0/55987853 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fca24103530 0x7fca24198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.321+0000 7fca28bd0640 1 --2- 172.21.15.46:0/55987853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca24100c00 0x7fca24197e50 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.321+0000 7fca28bd0640 1 -- 172.21.15.46:0/55987853 >> 172.21.15.46:0/55987853 conn(0x7fca240fa550 msgr2=0x7fca240ffa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:52.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.321+0000 7fca28bd0640 1 -- 172.21.15.46:0/55987853 shutdown_connections 2024-04-27T04:01:52.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.321+0000 7fca28bd0640 1 -- 172.21.15.46:0/55987853 wait complete. 2024-04-27T04:01:52.331 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:52 smithi046 ceph-mon[26899]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 8.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-27T04:01:52.331 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:52 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:52.332 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:52 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:52.332 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:52 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:01:52.332 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:52 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:01:52.332 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:52 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:52.332 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:52 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/9959520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T04:01:52.332 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:52 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2013819789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T04:01:52.332 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:52 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/4167041233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-27T04:01:52.333 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-27T04:01:52.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.433+0000 7fb87cfeb640 1 -- 172.21.15.46:0/3622598238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb878100b80 msgr2=0x7fb878102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:52.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.433+0000 7fb87cfeb640 1 --2- 172.21.15.46:0/3622598238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb878100b80 0x7fb878102f70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb868009c50 tx=0x7fb86802c190 comp rx=0 tx=0).stop 2024-04-27T04:01:52.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.434+0000 7fb87cfeb640 1 -- 172.21.15.46:0/3622598238 shutdown_connections 2024-04-27T04:01:52.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.434+0000 7fb87cfeb640 1 --2- 172.21.15.46:0/3622598238 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb8781034b0 0x7fb8781058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.434+0000 7fb87cfeb640 1 --2- 172.21.15.46:0/3622598238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb878100b80 0x7fb878102f70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.434+0000 7fb87cfeb640 1 -- 172.21.15.46:0/3622598238 >> 172.21.15.46:0/3622598238 conn(0x7fb8780fa590 msgr2=0x7fb8780fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:52.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.434+0000 7fb87cfeb640 1 -- 172.21.15.46:0/3622598238 shutdown_connections 2024-04-27T04:01:52.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.434+0000 7fb87cfeb640 1 -- 172.21.15.46:0/3622598238 wait complete. 2024-04-27T04:01:52.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.434+0000 7fb87cfeb640 1 Processor -- start 2024-04-27T04:01:52.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.434+0000 7fb87cfeb640 1 -- start start 2024-04-27T04:01:52.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb87cfeb640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb878100b80 0x7fb878195c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:52.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb87cfeb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8781034b0 0x7fb878196170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:52.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb87cfeb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb878196740 con 0x7fb8781034b0 2024-04-27T04:01:52.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb87cfeb640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8781968b0 con 0x7fb878100b80 2024-04-27T04:01:52.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb875d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8781034b0 0x7fb878196170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:52.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb875d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8781034b0 0x7fb878196170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38856/0 (socket says 172.21.15.46:38856) 2024-04-27T04:01:52.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb875d9b640 1 -- 172.21.15.46:0/539225506 learned_addr learned my addr 172.21.15.46:0/539225506 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:52.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb87659c640 1 --2- 172.21.15.46:0/539225506 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb878100b80 0x7fb878195c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:52.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb875d9b640 1 -- 172.21.15.46:0/539225506 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb878100b80 msgr2=0x7fb878195c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:52.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb875d9b640 1 --2- 172.21.15.46:0/539225506 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb878100b80 0x7fb878195c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb875d9b640 1 -- 172.21.15.46:0/539225506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb868009900 con 0x7fb8781034b0 2024-04-27T04:01:52.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb87659c640 1 --2- 172.21.15.46:0/539225506 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb878100b80 0x7fb878195c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T04:01:52.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb875d9b640 1 --2- 172.21.15.46:0/539225506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8781034b0 0x7fb878196170 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb86c00d6c0 tx=0x7fb86c00db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:52.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb8677fe640 1 -- 172.21.15.46:0/539225506 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb86c0051f0 con 0x7fb8781034b0 2024-04-27T04:01:52.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb8677fe640 1 -- 172.21.15.46:0/539225506 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb86c009ce0 con 0x7fb8781034b0 2024-04-27T04:01:52.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb87cfeb640 1 -- 172.21.15.46:0/539225506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87819b2f0 con 0x7fb8781034b0 2024-04-27T04:01:52.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb8677fe640 1 -- 172.21.15.46:0/539225506 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb86c010460 con 0x7fb8781034b0 2024-04-27T04:01:52.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.435+0000 7fb87cfeb640 1 -- 172.21.15.46:0/539225506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87819b850 con 0x7fb8781034b0 2024-04-27T04:01:52.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.436+0000 7fb87cfeb640 1 -- 172.21.15.46:0/539225506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb878108d20 con 0x7fb8781034b0 2024-04-27T04:01:52.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.437+0000 7fb8677fe640 1 -- 172.21.15.46:0/539225506 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb86c005350 con 0x7fb8781034b0 2024-04-27T04:01:52.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.437+0000 7fb8677fe640 1 --2- 172.21.15.46:0/539225506 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb86006cc50 0x7fb86006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:52.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.438+0000 7fb87659c640 1 --2- 172.21.15.46:0/539225506 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb86006cc50 0x7fb86006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:52.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.438+0000 7fb8677fe640 1 -- 172.21.15.46:0/539225506 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb86c08bda0 con 0x7fb8781034b0 2024-04-27T04:01:52.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.438+0000 7fb87659c640 1 --2- 172.21.15.46:0/539225506 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb86006cc50 0x7fb86006f110 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb868006010 tx=0x7fb868009800 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:52.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.442+0000 7fb8677fe640 1 -- 172.21.15.46:0/539225506 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fb86c056110 con 0x7fb8781034b0 2024-04-27T04:01:52.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:52 smithi073 ceph-mon[34160]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 8.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-27T04:01:52.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:52 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:52.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:52 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:52.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:52 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:01:52.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:52 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:01:52.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:52 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:01:52.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:52 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/9959520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T04:01:52.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:52 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/2013819789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T04:01:52.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:52 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/4167041233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-27T04:01:52.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.662+0000 7fb87cfeb640 1 -- 172.21.15.46:0/539225506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb87819ba70 con 0x7fb8781034b0 2024-04-27T04:01:52.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.663+0000 7fb8677fe640 1 -- 172.21.15.46:0/539225506 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4853 (secure 0 0 0) 0x7fb86c059730 con 0x7fb8781034b0 2024-04-27T04:01:52.663 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:01:52.664 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-27T04:01:21.676811+0000","modified":"2024-04-27T04:01:42.132288+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14488,"mds_1":24285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14488":{"gid":14488,"name":"cephfs.smithi046.mycakd","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.46:6829/3081610305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3081610305},{"type":"v1","addr":"172.21.15.46:6829","nonce":3081610305}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24285":{"gid":24285,"name":"cephfs.smithi073.zkoypj","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.73:6825/3397960510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":3397960510},{"type":"v1","addr":"172.21.15.73:6825","nonce":3397960510}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24295":{"gid":24295,"name":"cephfs.smithi073.yryono","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.73:6827/565262827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":565262827},{"type":"v1","addr":"172.21.15.73:6827","nonce":565262827}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24303":{"gid":24303,"name":"cephfs.smithi046.qgmaql","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6827/2653255940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2653255940},{"type":"v1","addr":"172.21.15.46:6827","nonce":2653255940}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-27T04:01:52.664 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 19 2024-04-27T04:01:52.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.665+0000 7fb87cfeb640 1 -- 172.21.15.46:0/539225506 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb86006cc50 msgr2=0x7fb86006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:52.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.665+0000 7fb87cfeb640 1 --2- 172.21.15.46:0/539225506 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb86006cc50 0x7fb86006f110 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb868006010 tx=0x7fb868009800 comp rx=0 tx=0).stop 2024-04-27T04:01:52.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.665+0000 7fb87cfeb640 1 -- 172.21.15.46:0/539225506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8781034b0 msgr2=0x7fb878196170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:52.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.665+0000 7fb87cfeb640 1 --2- 172.21.15.46:0/539225506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8781034b0 0x7fb878196170 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb86c00d6c0 tx=0x7fb86c00db80 comp rx=0 tx=0).stop 2024-04-27T04:01:52.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.665+0000 7fb87cfeb640 1 -- 172.21.15.46:0/539225506 shutdown_connections 2024-04-27T04:01:52.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.665+0000 7fb87cfeb640 1 --2- 172.21.15.46:0/539225506 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb86006cc50 0x7fb86006f110 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.665+0000 7fb87cfeb640 1 --2- 172.21.15.46:0/539225506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8781034b0 0x7fb878196170 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.665+0000 7fb87cfeb640 1 --2- 172.21.15.46:0/539225506 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb878100b80 0x7fb878195c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:52.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.665+0000 7fb87cfeb640 1 -- 172.21.15.46:0/539225506 >> 172.21.15.46:0/539225506 conn(0x7fb8780fa590 msgr2=0x7fb8780fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:52.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.665+0000 7fb87cfeb640 1 -- 172.21.15.46:0/539225506 shutdown_connections 2024-04-27T04:01:52.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:52.665+0000 7fb87cfeb640 1 -- 172.21.15.46:0/539225506 wait complete. 2024-04-27T04:01:52.677 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 19, 'max_mds': 2, 'flags': 50} 2024-04-27T04:01:52.677 INFO:teuthology.run_tasks:Running task kclient... 2024-04-27T04:01:52.800 INFO:tasks.kclient:Mounting kernel clients... 2024-04-27T04:01:52.801 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-04-27T04:01:52.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:01:52.801 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-04-27T04:01:52.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:01:52.818 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2024-04-27T04:01:52.884 INFO:teuthology.orchestra.run.smithi046.stderr:Cannot find device "ceph-brx" 2024-04-27T04:01:52.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T04:01:52.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:01:52.886 DEBUG:teuthology.orchestra.run.smithi073:> ip netns list 2024-04-27T04:01:52.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:01:52.905 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link delete ceph-brx 2024-04-27T04:01:52.978 INFO:teuthology.orchestra.run.smithi073.stderr:Cannot find device "ceph-brx" 2024-04-27T04:01:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T04:01:52.980 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-04-27T04:01:52.980 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-27T04:01:52.980 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-27T04:01:52.980 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-27T04:01:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:01:53.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:53 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/55987853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-27T04:01:53.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:53 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/539225506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T04:01:53.584 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:53 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/55987853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-27T04:01:53.584 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:53 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/539225506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T04:01:54.521 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:54 smithi046 ceph-mon[26899]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-27T04:01:54.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:54 smithi073 ceph-mon[34160]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-27T04:01:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.738+0000 7f194854c700 1 -- 172.21.15.46:0/1982083158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1940101ce0 msgr2=0x7f19401020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.738+0000 7f194854c700 1 --2- 172.21.15.46:0/1982083158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1940101ce0 0x7f19401020c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f193c009910 tx=0x7f193c009c20 comp rx=0 tx=0).stop 2024-04-27T04:01:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.738+0000 7f194854c700 1 -- 172.21.15.46:0/1982083158 shutdown_connections 2024-04-27T04:01:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.738+0000 7f194854c700 1 --2- 172.21.15.46:0/1982083158 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1940102690 0x7f194010ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.738+0000 7f194854c700 1 --2- 172.21.15.46:0/1982083158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1940101ce0 0x7f19401020c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.738+0000 7f194854c700 1 -- 172.21.15.46:0/1982083158 >> 172.21.15.46:0/1982083158 conn(0x7f19400fb5e0 msgr2=0x7f19400fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.738+0000 7f194854c700 1 -- 172.21.15.46:0/1982083158 shutdown_connections 2024-04-27T04:01:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.738+0000 7f194854c700 1 -- 172.21.15.46:0/1982083158 wait complete. 2024-04-27T04:01:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.738+0000 7f194854c700 1 Processor -- start 2024-04-27T04:01:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.738+0000 7f194854c700 1 -- start start 2024-04-27T04:01:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.738+0000 7f194854c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1940101ce0 0x7f194019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.738+0000 7f194854c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1940102690 0x7f194019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.738+0000 7f194854c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194019d380 con 0x7f1940101ce0 2024-04-27T04:01:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.738+0000 7f194854c700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19401969b0 con 0x7f1940102690 2024-04-27T04:01:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.739+0000 7f19462e8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1940101ce0 0x7f194019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.739+0000 7f19462e8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1940101ce0 0x7f194019c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38870/0 (socket says 172.21.15.46:38870) 2024-04-27T04:01:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.739+0000 7f19462e8700 1 -- 172.21.15.46:0/3520723079 learned_addr learned my addr 172.21.15.46:0/3520723079 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.739+0000 7f1945ae7700 1 --2- 172.21.15.46:0/3520723079 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1940102690 0x7f194019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.739+0000 7f19462e8700 1 -- 172.21.15.46:0/3520723079 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1940102690 msgr2=0x7f194019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.739+0000 7f19462e8700 1 --2- 172.21.15.46:0/3520723079 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1940102690 0x7f194019cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.739+0000 7f19462e8700 1 -- 172.21.15.46:0/3520723079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c0095f0 con 0x7f1940101ce0 2024-04-27T04:01:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.739+0000 7f1945ae7700 1 --2- 172.21.15.46:0/3520723079 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1940102690 0x7f194019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T04:01:54.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.739+0000 7f19462e8700 1 --2- 172.21.15.46:0/3520723079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1940101ce0 0x7f194019c820 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f193c000c00 tx=0x7f193c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:54.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.739+0000 7f19377fe700 1 -- 172.21.15.46:0/3520723079 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f193c01d070 con 0x7f1940101ce0 2024-04-27T04:01:54.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.739+0000 7f19377fe700 1 -- 172.21.15.46:0/3520723079 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f193c0056e0 con 0x7f1940101ce0 2024-04-27T04:01:54.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.739+0000 7f194854c700 1 -- 172.21.15.46:0/3520723079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1940196c30 con 0x7f1940101ce0 2024-04-27T04:01:54.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.739+0000 7f19377fe700 1 -- 172.21.15.46:0/3520723079 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f193c00fe40 con 0x7f1940101ce0 2024-04-27T04:01:54.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.739+0000 7f194854c700 1 -- 172.21.15.46:0/3520723079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1940197120 con 0x7f1940101ce0 2024-04-27T04:01:54.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.740+0000 7f194854c700 1 -- 172.21.15.46:0/3520723079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1940105500 con 0x7f1940101ce0 2024-04-27T04:01:54.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.741+0000 7f19377fe700 1 -- 172.21.15.46:0/3520723079 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f193c00f680 con 0x7f1940101ce0 2024-04-27T04:01:54.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.741+0000 7f19377fe700 1 --2- 172.21.15.46:0/3520723079 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f192c06cc90 0x7f192c06f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:54.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.742+0000 7f1945ae7700 1 --2- 172.21.15.46:0/3520723079 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f192c06cc90 0x7f192c06f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:54.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.742+0000 7f19377fe700 1 -- 172.21.15.46:0/3520723079 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f193c08d6e0 con 0x7f1940101ce0 2024-04-27T04:01:54.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.742+0000 7f1945ae7700 1 --2- 172.21.15.46:0/3520723079 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f192c06cc90 0x7f192c06f150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1940197b20 tx=0x7f193000b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:54.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.746+0000 7f19377fe700 1 -- 172.21.15.46:0/3520723079 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f193c05b440 con 0x7f1940101ce0 2024-04-27T04:01:54.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.998+0000 7f194854c700 1 -- 172.21.15.46:0/3520723079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1940197e20 con 0x7f1940101ce0 2024-04-27T04:01:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:54.998+0000 7f19377fe700 1 -- 172.21.15.46:0/3520723079 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+83 (secure 0 0 0) 0x7f193c05afd0 con 0x7f1940101ce0 2024-04-27T04:01:54.999 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-27T04:01:55.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:55.001+0000 7f194854c700 1 -- 172.21.15.46:0/3520723079 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f192c06cc90 msgr2=0x7f192c06f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:55.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:55.001+0000 7f194854c700 1 --2- 172.21.15.46:0/3520723079 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f192c06cc90 0x7f192c06f150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1940197b20 tx=0x7f193000b410 comp rx=0 tx=0).stop 2024-04-27T04:01:55.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:55.001+0000 7f194854c700 1 -- 172.21.15.46:0/3520723079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1940101ce0 msgr2=0x7f194019c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:55.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:55.001+0000 7f194854c700 1 --2- 172.21.15.46:0/3520723079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1940101ce0 0x7f194019c820 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f193c000c00 tx=0x7f193c00bd70 comp rx=0 tx=0).stop 2024-04-27T04:01:55.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:55.001+0000 7f194854c700 1 -- 172.21.15.46:0/3520723079 shutdown_connections 2024-04-27T04:01:55.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:55.001+0000 7f194854c700 1 --2- 172.21.15.46:0/3520723079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1940101ce0 0x7f194019c820 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:55.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:55.001+0000 7f194854c700 1 --2- 172.21.15.46:0/3520723079 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f192c06cc90 0x7f192c06f150 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:55.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:55.001+0000 7f194854c700 1 --2- 172.21.15.46:0/3520723079 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1940102690 0x7f194019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:55.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:55.001+0000 7f194854c700 1 -- 172.21.15.46:0/3520723079 >> 172.21.15.46:0/3520723079 conn(0x7f19400fb5e0 msgr2=0x7f1940103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:55.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:55.001+0000 7f194854c700 1 -- 172.21.15.46:0/3520723079 shutdown_connections 2024-04-27T04:01:55.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:55.001+0000 7f194854c700 1 -- 172.21.15.46:0/3520723079 wait complete. 2024-04-27T04:01:55.562 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:55 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/3520723079' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-27T04:01:55.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:55 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/3520723079' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-27T04:01:55.585 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-27T04:01:55.585 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-27T04:01:55.585 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-04-27T04:01:55.586 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-27T04:01:55.586 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-27T04:01:55.586 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-27T04:01:55.586 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-27T04:01:55.586 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-27T04:01:55.586 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-27T04:01:55.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:01:55.586 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-04-27T04:01:55.603 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-27T04:01:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-27T04:01:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-04-27T04:01:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-27T04:01:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-04-27T04:01:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-27T04:01:55.603 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-27T04:01:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-04-27T04:01:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-04-27T04:01:55.604 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-27T04:01:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-04-27T04:01:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-04-27T04:01:55.604 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-27T04:01:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-04-27T04:01:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-04-27T04:01:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-27T04:01:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42105sec preferred_lft 42105sec 2024-04-27T04:01:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-04-27T04:01:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-27T04:01:55.604 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-27T04:01:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-04-27T04:01:55.605 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-04-27T04:01:55.605 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-27T04:01:55.605 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T04:01:55.605 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-27T04:01:55.605 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add name ceph-brx type bridge 2024-04-27T04:01:55.605 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr flush dev ceph-brx 2024-04-27T04:01:55.605 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set ceph-brx up 2024-04-27T04:01:55.605 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-27T04:01:55.605 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-27T04:01:55.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T04:01:55.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:01:55.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:01:55.790 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-27T04:01:55.815 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-04-27T04:01:55.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:01:55.817 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-04-27T04:01:55.890 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-04-27T04:01:55.890 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-27T04:01:55.891 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-27T04:01:55.891 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-27T04:01:55.891 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-27T04:01:55.891 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T04:01:55.891 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-27T04:01:55.891 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-27T04:01:55.891 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-27T04:01:55.891 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-27T04:01:55.892 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-27T04:01:55.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T04:01:56.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:01:56.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:01:56.174 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-04-27T04:01:56.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:01:56.190 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2024-04-27T04:01:56.244 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T04:01:56.244 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-27T04:01:56.244 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-27T04:01:56.244 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-27T04:01:56.245 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-27T04:01:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T04:01:56.340 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:56 smithi046 ceph-mon[26899]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-27T04:01:56.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:01:56.362 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-27T04:01:56.362 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T04:01:56.362 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-27T04:01:56.362 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-27T04:01:56.363 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-27T04:01:56.363 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-27T04:01:56.363 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-27T04:01:56.363 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-27T04:01:56.363 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-27T04:01:56.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T04:01:56.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:01:56.488 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T04:01:56.488 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-27T04:01:56.488 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-04-27T04:01:56.488 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.0 master ceph-brx 2024-04-27T04:01:56.488 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-27T04:01:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T04:01:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:01:56.565 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-27T04:01:56.565 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-27T04:01:56.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:56 smithi073 ceph-mon[34160]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-27T04:01:56.580 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-27T04:01:56.580 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-27T04:01:56.581 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-27T04:01:56.631 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-27T04:01:56.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:01:56.631 DEBUG:teuthology.orchestra.run.smithi046:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-27T04:01:56.717 INFO:teuthology.orchestra.run.smithi046.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-27T04:01:56.717 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-27T04:01:56.717 INFO:teuthology.orchestra.run.smithi046.stdout:invalid new device string format 2024-04-27T04:01:56.819 INFO:teuthology.orchestra.run.smithi046.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-27T04:01:56.819 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-27T04:01:56.819 INFO:teuthology.orchestra.run.smithi046.stdout:invalid new device string format 2024-04-27T04:01:56.819 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: resolved to: "172.21.15.46:3300,172.21.15.73:3300" 2024-04-27T04:01:56.819 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-27T04:01:56.825 INFO:teuthology.orchestra.run.smithi046.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-27T04:01:56.825 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-27T04:01:56.825 INFO:teuthology.orchestra.run.smithi046.stdout:invalid new device string format 2024-04-27T04:01:56.825 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: resolved to: "172.21.15.46:3300,172.21.15.73:3300" 2024-04-27T04:01:56.825 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-27T04:01:56.825 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: resolved to: "172.21.15.46:6789,172.21.15.73:6789" 2024-04-27T04:01:56.825 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.46:6789,172.21.15.73:6789:/ 2024-04-27T04:01:56.825 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=b1d3a2d0-0449-11ef-bc93-c7b262605968" will pass to kernel 2024-04-27T04:01:56.826 INFO:teuthology.orchestra.run.smithi046.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-27T04:01:56.826 INFO:teuthology.orchestra.run.smithi046.stdout: You just mounted a file system that supports labels which does not 2024-04-27T04:01:56.826 INFO:teuthology.orchestra.run.smithi046.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-27T04:01:56.826 INFO:teuthology.orchestra.run.smithi046.stdout: applications will generate AVC messages and not be allowed access to 2024-04-27T04:01:56.826 INFO:teuthology.orchestra.run.smithi046.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-27T04:01:56.828 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-27T04:01:56.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:01:56.828 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-27T04:01:56.856 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-27T04:01:56.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-27T04:01:56.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:01:56.958 INFO:teuthology.orchestra.run.smithi046.stdout:client14538 2024-04-27T04:01:56.958 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T04:01:56.959 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T04:01:56.976 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-27T04:01:57.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-27T04:01:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:01:57.078 INFO:teuthology.orchestra.run.smithi046.stdout:client14538 2024-04-27T04:01:57.078 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-27T04:01:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-27T04:01:57.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:01:57.139 INFO:teuthology.orchestra.run.smithi046.stdout:client14538 2024-04-27T04:01:57.139 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968.client14538/status') 2024-04-27T04:01:57.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T04:01:57.202 INFO:teuthology.orchestra.run.smithi046.stdout:instance: client.14538 (3)192.168.0.1:0/3367561359 2024-04-27T04:01:57.202 INFO:teuthology.orchestra.run.smithi046.stdout:blocklisted: false 2024-04-27T04:01:57.202 INFO:teuthology.orchestra.run.smithi046.stderr:0+1 records in 2024-04-27T04:01:57.202 INFO:teuthology.orchestra.run.smithi046.stderr:0+1 records out 2024-04-27T04:01:57.202 INFO:teuthology.orchestra.run.smithi046.stderr:70 bytes copied, 2.3472e-05 s, 3.0 MB/s 2024-04-27T04:01:57.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:01:57.210 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-04-27T04:01:57.210 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-27T04:01:57.210 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-27T04:01:57.210 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-27T04:01:57.422 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:01:58.546 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:01:58 smithi046 ceph-mon[26899]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-27T04:01:58.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:01:58 smithi073 ceph-mon[34160]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-27T04:01:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.046+0000 7fe171606700 1 -- 172.21.15.46:0/1292404591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c102760 msgr2=0x7fe16c10ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.046+0000 7fe171606700 1 --2- 172.21.15.46:0/1292404591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c102760 0x7fe16c10ac50 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe160009910 tx=0x7fe160009c20 comp rx=0 tx=0).stop 2024-04-27T04:01:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.046+0000 7fe171606700 1 -- 172.21.15.46:0/1292404591 shutdown_connections 2024-04-27T04:01:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.046+0000 7fe171606700 1 --2- 172.21.15.46:0/1292404591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c102760 0x7fe16c10ac50 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.046+0000 7fe171606700 1 --2- 172.21.15.46:0/1292404591 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe16c101e40 0x7fe16c102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.046+0000 7fe171606700 1 -- 172.21.15.46:0/1292404591 >> 172.21.15.46:0/1292404591 conn(0x7fe16c0fb5f0 msgr2=0x7fe16c0fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.046+0000 7fe171606700 1 -- 172.21.15.46:0/1292404591 shutdown_connections 2024-04-27T04:01:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.046+0000 7fe171606700 1 -- 172.21.15.46:0/1292404591 wait complete. 2024-04-27T04:01:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.046+0000 7fe171606700 1 Processor -- start 2024-04-27T04:01:59.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.047+0000 7fe171606700 1 -- start start 2024-04-27T04:01:59.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.047+0000 7fe171606700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c101e40 0x7fe16c0ffaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:59.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.047+0000 7fe171606700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe16c102760 0x7fe16c0fffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:59.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.047+0000 7fe171606700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c100520 con 0x7fe16c101e40 2024-04-27T04:01:59.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.047+0000 7fe171606700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c100660 con 0x7fe16c102760 2024-04-27T04:01:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.047+0000 7fe16affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c101e40 0x7fe16c0ffaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.047+0000 7fe16affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c101e40 0x7fe16c0ffaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38892/0 (socket says 172.21.15.46:38892) 2024-04-27T04:01:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.047+0000 7fe16affd700 1 -- 172.21.15.46:0/262163030 learned_addr learned my addr 172.21.15.46:0/262163030 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:01:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.047+0000 7fe16a7fc700 1 --2- 172.21.15.46:0/262163030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe16c102760 0x7fe16c0fffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.047+0000 7fe16affd700 1 -- 172.21.15.46:0/262163030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe16c102760 msgr2=0x7fe16c0fffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.047+0000 7fe16affd700 1 --2- 172.21.15.46:0/262163030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe16c102760 0x7fe16c0fffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.047+0000 7fe16affd700 1 -- 172.21.15.46:0/262163030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1600095f0 con 0x7fe16c101e40 2024-04-27T04:01:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.048+0000 7fe16affd700 1 --2- 172.21.15.46:0/262163030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c101e40 0x7fe16c0ffaa0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe15c00b4b0 tx=0x7fe15c00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.048+0000 7fe153fff700 1 -- 172.21.15.46:0/262163030 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe15c011800 con 0x7fe16c101e40 2024-04-27T04:01:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.048+0000 7fe153fff700 1 -- 172.21.15.46:0/262163030 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe15c011e20 con 0x7fe16c101e40 2024-04-27T04:01:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.048+0000 7fe171606700 1 -- 172.21.15.46:0/262163030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16c1a2df0 con 0x7fe16c101e40 2024-04-27T04:01:59.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.048+0000 7fe153fff700 1 -- 172.21.15.46:0/262163030 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe15c00f2e0 con 0x7fe16c101e40 2024-04-27T04:01:59.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.048+0000 7fe171606700 1 -- 172.21.15.46:0/262163030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16c1a3150 con 0x7fe16c101e40 2024-04-27T04:01:59.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.049+0000 7fe171606700 1 -- 172.21.15.46:0/262163030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe16c069930 con 0x7fe16c101e40 2024-04-27T04:01:59.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.049+0000 7fe153fff700 1 -- 172.21.15.46:0/262163030 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe15c010380 con 0x7fe16c101e40 2024-04-27T04:01:59.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.050+0000 7fe153fff700 1 --2- 172.21.15.46:0/262163030 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe15406cce0 0x7fe15406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:01:59.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.050+0000 7fe16a7fc700 1 --2- 172.21.15.46:0/262163030 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe15406cce0 0x7fe15406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:01:59.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.050+0000 7fe153fff700 1 -- 172.21.15.46:0/262163030 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fe15c08bdd0 con 0x7fe16c101e40 2024-04-27T04:01:59.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.050+0000 7fe16a7fc700 1 --2- 172.21.15.46:0/262163030 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe15406cce0 0x7fe15406f1a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe16c1011d0 tx=0x7fe16001a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:01:59.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.055+0000 7fe153fff700 1 -- 172.21.15.46:0/262163030 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fe15c056140 con 0x7fe16c101e40 2024-04-27T04:01:59.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.306+0000 7fe171606700 1 -- 172.21.15.46:0/262163030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe16c100820 con 0x7fe16c101e40 2024-04-27T04:01:59.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.306+0000 7fe153fff700 1 -- 172.21.15.46:0/262163030 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+83 (secure 0 0 0) 0x7fe15c059760 con 0x7fe16c101e40 2024-04-27T04:01:59.307 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-27T04:01:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.308+0000 7fe171606700 1 -- 172.21.15.46:0/262163030 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe15406cce0 msgr2=0x7fe15406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.308+0000 7fe171606700 1 --2- 172.21.15.46:0/262163030 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe15406cce0 0x7fe15406f1a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe16c1011d0 tx=0x7fe16001a040 comp rx=0 tx=0).stop 2024-04-27T04:01:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.308+0000 7fe171606700 1 -- 172.21.15.46:0/262163030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c101e40 msgr2=0x7fe16c0ffaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:01:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.308+0000 7fe171606700 1 --2- 172.21.15.46:0/262163030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c101e40 0x7fe16c0ffaa0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe15c00b4b0 tx=0x7fe15c00b7c0 comp rx=0 tx=0).stop 2024-04-27T04:01:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.309+0000 7fe171606700 1 -- 172.21.15.46:0/262163030 shutdown_connections 2024-04-27T04:01:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.309+0000 7fe171606700 1 --2- 172.21.15.46:0/262163030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c101e40 0x7fe16c0ffaa0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.309+0000 7fe171606700 1 --2- 172.21.15.46:0/262163030 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe15406cce0 0x7fe15406f1a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.309+0000 7fe171606700 1 --2- 172.21.15.46:0/262163030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe16c102760 0x7fe16c0fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:01:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.309+0000 7fe171606700 1 -- 172.21.15.46:0/262163030 >> 172.21.15.46:0/262163030 conn(0x7fe16c0fb5f0 msgr2=0x7fe16c103130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:01:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.309+0000 7fe171606700 1 -- 172.21.15.46:0/262163030 shutdown_connections 2024-04-27T04:01:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:01:59.309+0000 7fe171606700 1 -- 172.21.15.46:0/262163030 wait complete. 2024-04-27T04:01:59.959 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-27T04:01:59.959 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-27T04:01:59.959 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi073.front.sepia.ceph.com 2024-04-27T04:01:59.959 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-27T04:01:59.959 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-27T04:01:59.959 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-27T04:01:59.959 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-27T04:01:59.959 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-27T04:01:59.959 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-27T04:01:59.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:01:59.960 DEBUG:teuthology.orchestra.run.smithi073:> ip addr 2024-04-27T04:01:59.980 INFO:teuthology.orchestra.run.smithi073.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-27T04:01:59.980 INFO:teuthology.orchestra.run.smithi073.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-27T04:01:59.980 INFO:teuthology.orchestra.run.smithi073.stdout: inet 127.0.0.1/8 scope host lo 2024-04-27T04:01:59.980 INFO:teuthology.orchestra.run.smithi073.stdout: valid_lft forever preferred_lft forever 2024-04-27T04:01:59.980 INFO:teuthology.orchestra.run.smithi073.stdout: inet6 ::1/128 scope host 2024-04-27T04:01:59.981 INFO:teuthology.orchestra.run.smithi073.stdout: valid_lft forever preferred_lft forever 2024-04-27T04:01:59.981 INFO:teuthology.orchestra.run.smithi073.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-27T04:01:59.981 INFO:teuthology.orchestra.run.smithi073.stdout: link/ether 0c:c4:7a:d8:d9:ea brd ff:ff:ff:ff:ff:ff 2024-04-27T04:01:59.981 INFO:teuthology.orchestra.run.smithi073.stdout: altname enp7s0f0 2024-04-27T04:01:59.981 INFO:teuthology.orchestra.run.smithi073.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-27T04:01:59.981 INFO:teuthology.orchestra.run.smithi073.stdout: link/ether 0c:c4:7a:d8:d9:eb brd ff:ff:ff:ff:ff:ff 2024-04-27T04:01:59.981 INFO:teuthology.orchestra.run.smithi073.stdout: altname enp7s0f1 2024-04-27T04:01:59.981 INFO:teuthology.orchestra.run.smithi073.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-27T04:01:59.981 INFO:teuthology.orchestra.run.smithi073.stdout: link/ether 0c:c4:7a:8f:ce:7c brd ff:ff:ff:ff:ff:ff 2024-04-27T04:01:59.981 INFO:teuthology.orchestra.run.smithi073.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-27T04:01:59.981 INFO:teuthology.orchestra.run.smithi073.stdout: link/ether 0c:c4:7a:8f:ce:7d brd ff:ff:ff:ff:ff:ff 2024-04-27T04:01:59.981 INFO:teuthology.orchestra.run.smithi073.stdout: inet 172.21.15.73/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-27T04:01:59.981 INFO:teuthology.orchestra.run.smithi073.stdout: valid_lft 42093sec preferred_lft 42093sec 2024-04-27T04:01:59.982 INFO:teuthology.orchestra.run.smithi073.stdout: inet6 fe80::8279:384d:3150:5fd/64 scope link noprefixroute 2024-04-27T04:01:59.982 INFO:teuthology.orchestra.run.smithi073.stdout: valid_lft forever preferred_lft forever 2024-04-27T04:01:59.982 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-27T04:01:59.982 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T04:01:59.982 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-04-27T04:01:59.982 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link add name ceph-brx type bridge 2024-04-27T04:01:59.982 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip addr flush dev ceph-brx 2024-04-27T04:01:59.982 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link set ceph-brx up 2024-04-27T04:01:59.982 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-27T04:01:59.982 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-04-27T04:02:00.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T04:02:00.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:02:00.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:02:00.174 DEBUG:teuthology.orchestra.run.smithi073:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-27T04:02:00.197 INFO:teuthology.orchestra.run.smithi073.stdout:1 2024-04-27T04:02:00.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:02:00.198 DEBUG:teuthology.orchestra.run.smithi073:> route 2024-04-27T04:02:00.267 INFO:teuthology.orchestra.run.smithi073.stdout:Kernel IP routing table 2024-04-27T04:02:00.267 INFO:teuthology.orchestra.run.smithi073.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-27T04:02:00.268 INFO:teuthology.orchestra.run.smithi073.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-27T04:02:00.268 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-27T04:02:00.268 INFO:teuthology.orchestra.run.smithi073.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-27T04:02:00.268 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T04:02:00.268 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-04-27T04:02:00.268 DEBUG:teuthology.orchestra.run.smithi073:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-27T04:02:00.268 DEBUG:teuthology.orchestra.run.smithi073:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-27T04:02:00.268 DEBUG:teuthology.orchestra.run.smithi073:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-27T04:02:00.269 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-04-27T04:02:00.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T04:02:00.362 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:00 smithi073 ceph-mon[34160]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-27T04:02:00.362 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:00 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/262163030' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-27T04:02:00.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:02:00.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:02:00.559 DEBUG:teuthology.orchestra.run.smithi073:> ip netns list 2024-04-27T04:02:00.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:02:00.574 DEBUG:teuthology.orchestra.run.smithi073:> ip netns list-id 2024-04-27T04:02:00.629 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T04:02:00.629 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-04-27T04:02:00.629 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-27T04:02:00.629 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-27T04:02:00.629 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-04-27T04:02:00.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:00 smithi046 ceph-mon[26899]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-27T04:02:00.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:00 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/262163030' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-27T04:02:00.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T04:02:00.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:02:00.744 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-27T04:02:00.744 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T04:02:00.744 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-04-27T04:02:00.744 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-27T04:02:00.744 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-27T04:02:00.744 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-27T04:02:00.744 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-27T04:02:00.744 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-27T04:02:00.744 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-04-27T04:02:00.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T04:02:00.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:02:00.861 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T04:02:00.861 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-04-27T04:02:00.861 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link set brx.0 up 2024-04-27T04:02:00.862 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link set dev brx.0 master ceph-brx 2024-04-27T04:02:00.862 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-04-27T04:02:00.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T04:02:00.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:02:00.936 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-27T04:02:00.936 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-27T04:02:00.951 INFO:teuthology.orchestra.run.smithi073.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-27T04:02:00.951 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-27T04:02:00.952 DEBUG:teuthology.orchestra.run.smithi073:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-27T04:02:01.006 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-27T04:02:01.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:02:01.006 DEBUG:teuthology.orchestra.run.smithi073:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-27T04:02:01.095 INFO:teuthology.orchestra.run.smithi073.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-27T04:02:01.095 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-27T04:02:01.095 INFO:teuthology.orchestra.run.smithi073.stdout:invalid new device string format 2024-04-27T04:02:01.196 INFO:teuthology.orchestra.run.smithi073.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-27T04:02:01.196 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-27T04:02:01.196 INFO:teuthology.orchestra.run.smithi073.stdout:invalid new device string format 2024-04-27T04:02:01.196 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: resolved to: "172.21.15.46:3300,172.21.15.73:3300" 2024-04-27T04:02:01.197 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-27T04:02:01.201 INFO:teuthology.orchestra.run.smithi073.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-27T04:02:01.201 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-27T04:02:01.201 INFO:teuthology.orchestra.run.smithi073.stdout:invalid new device string format 2024-04-27T04:02:01.202 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: resolved to: "172.21.15.46:3300,172.21.15.73:3300" 2024-04-27T04:02:01.202 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-27T04:02:01.202 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: resolved to: "172.21.15.46:6789,172.21.15.73:6789" 2024-04-27T04:02:01.202 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.46:6789,172.21.15.73:6789:/ 2024-04-27T04:02:01.202 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=b1d3a2d0-0449-11ef-bc93-c7b262605968" will pass to kernel 2024-04-27T04:02:01.202 INFO:teuthology.orchestra.run.smithi073.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-27T04:02:01.203 INFO:teuthology.orchestra.run.smithi073.stdout: You just mounted a file system that supports labels which does not 2024-04-27T04:02:01.203 INFO:teuthology.orchestra.run.smithi073.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-27T04:02:01.203 INFO:teuthology.orchestra.run.smithi073.stdout: applications will generate AVC messages and not be allowed access to 2024-04-27T04:02:01.203 INFO:teuthology.orchestra.run.smithi073.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-27T04:02:01.204 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-27T04:02:01.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:02:01.204 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-27T04:02:01.228 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-27T04:02:01.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-27T04:02:01.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:02:01.323 INFO:teuthology.orchestra.run.smithi073.stdout:client14544 2024-04-27T04:02:01.324 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-04-27T04:02:01.324 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T04:02:01.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:01 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:02:01.340 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-27T04:02:01.428 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-27T04:02:01.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:02:01.435 INFO:teuthology.orchestra.run.smithi073.stdout:client14544 2024-04-27T04:02:01.435 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-27T04:02:01.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-27T04:02:01.485 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:02:01.490 INFO:teuthology.orchestra.run.smithi073.stdout:client14544 2024-04-27T04:02:01.491 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968.client14544/status') 2024-04-27T04:02:01.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T04:02:01.547 INFO:teuthology.orchestra.run.smithi073.stdout:instance: client.14544 (3)172.21.15.73:0/3267025493 2024-04-27T04:02:01.547 INFO:teuthology.orchestra.run.smithi073.stdout:blocklisted: false 2024-04-27T04:02:01.547 INFO:teuthology.orchestra.run.smithi073.stderr:0+1 records in 2024-04-27T04:02:01.548 INFO:teuthology.orchestra.run.smithi073.stderr:0+1 records out 2024-04-27T04:02:01.548 INFO:teuthology.orchestra.run.smithi073.stderr:71 bytes copied, 2.6724e-05 s, 2.7 MB/s 2024-04-27T04:02:01.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-04-27T04:02:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T04:02:01.554 INFO:teuthology.run_tasks:Running task print... 2024-04-27T04:02:01.567 INFO:teuthology.task.print:**** done client 2024-04-27T04:02:01.567 INFO:teuthology.run_tasks:Running task parallel... 2024-04-27T04:02:01.585 INFO:teuthology.task.parallel:starting parallel... 2024-04-27T04:02:01.585 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-27T04:02:01.586 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-27T04:02:01.586 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi046.front.sepia.ceph.com 2024-04-27T04:02:01.586 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -e sha1=4e213a58e787437ff508e6c890ccd5fd32dc96b5 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-04-27T04:02:01.587 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-27T04:02:01.587 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-04-27T04:02:01.590 INFO:tasks.workunit:Pulling workunits from ref 3067e9f133f954b642fdf21114a79edb58590948 2024-04-27T04:02:01.591 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-27T04:02:01.591 INFO:tasks.workunit:timeout=3h 2024-04-27T04:02:01.591 INFO:tasks.workunit:cleanup=True 2024-04-27T04:02:01.591 DEBUG:teuthology.orchestra.run.smithi046:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-27T04:02:01.592 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:01 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:02:01.613 INFO:teuthology.orchestra.run.smithi046.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-27T04:02:01.613 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-27T04:02:01.613 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 7fh/127d Inode: 1 Links: 2 2024-04-27T04:02:01.613 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T04:02:01.613 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-27T04:02:01.613 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-27T04:02:01.613 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-27 04:01:26.671301980 +0000 2024-04-27T04:02:01.614 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-27 04:01:56.851795178 +0000 2024-04-27T04:02:01.614 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-27 04:01:26.671301980 +0000 2024-04-27T04:02:01.614 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-27T04:02:01.614 DEBUG:teuthology.orchestra.run.smithi046:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-27T04:02:01.685 DEBUG:teuthology.orchestra.run.smithi073:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-27T04:02:01.701 INFO:teuthology.orchestra.run.smithi073.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-04-27T04:02:01.701 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-04-27T04:02:01.701 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 7bh/123d Inode: 1 Links: 3 2024-04-27T04:02:01.702 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T04:02:01.702 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-27T04:02:01.702 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-27T04:02:01.702 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-04-27 04:02:01.678744039 +0000 2024-04-27T04:02:01.702 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-04-27 04:02:01.678744039 +0000 2024-04-27T04:02:01.702 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-04-27 04:01:26.671301980 +0000 2024-04-27T04:02:01.702 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-04-27T04:02:01.702 DEBUG:teuthology.orchestra.run.smithi073:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-04-27T04:02:01.769 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3067e9f133f954b642fdf21114a79edb58590948 2024-04-27T04:02:01.770 DEBUG:teuthology.orchestra.run.smithi073:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 3067e9f133f954b642fdf21114a79edb58590948 2024-04-27T04:02:01.786 INFO:tasks.workunit.client.0.smithi046.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-27T04:02:01.825 INFO:tasks.workunit.client.1.smithi073.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-27T04:02:01.839 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:02:02.548 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:02 smithi073 ceph-mon[34160]: pgmap v199: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-04-27T04:02:02.659 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:02 smithi046 ceph-mon[26899]: pgmap v199: 65 pgs: 65 active+clean; 580 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-04-27T04:02:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.514+0000 7f95f64b8700 1 -- 172.21.15.46:0/874965347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f0100d90 msgr2=0x7f95f01011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.514+0000 7f95f64b8700 1 --2- 172.21.15.46:0/874965347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f0100d90 0x7f95f01011f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f95d8009910 tx=0x7f95d8009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.514+0000 7f95f64b8700 1 -- 172.21.15.46:0/874965347 shutdown_connections 2024-04-27T04:02:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.514+0000 7f95f64b8700 1 --2- 172.21.15.46:0/874965347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f0100d90 0x7f95f01011f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.514+0000 7f95f64b8700 1 --2- 172.21.15.46:0/874965347 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f95f00ffb90 0x7f95f00fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.514+0000 7f95f64b8700 1 -- 172.21.15.46:0/874965347 >> 172.21.15.46:0/874965347 conn(0x7f95f00fb110 msgr2=0x7f95f00fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.514+0000 7f95f64b8700 1 -- 172.21.15.46:0/874965347 shutdown_connections 2024-04-27T04:02:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.514+0000 7f95f64b8700 1 -- 172.21.15.46:0/874965347 wait complete. 2024-04-27T04:02:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.514+0000 7f95f64b8700 1 Processor -- start 2024-04-27T04:02:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.514+0000 7f95f64b8700 1 -- start start 2024-04-27T04:02:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.515+0000 7f95f64b8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00ffb90 0x7f95f0196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.515+0000 7f95f64b8700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f95f0100d90 0x7f95f0196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.515+0000 7f95f64b8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f01971d0 con 0x7f95f00ffb90 2024-04-27T04:02:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.515+0000 7f95f64b8700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f0197310 con 0x7f95f0100d90 2024-04-27T04:02:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.515+0000 7f95effff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00ffb90 0x7f95f0196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.515+0000 7f95effff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00ffb90 0x7f95f0196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33298/0 (socket says 172.21.15.46:33298) 2024-04-27T04:02:03.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.515+0000 7f95effff700 1 -- 172.21.15.46:0/849957112 learned_addr learned my addr 172.21.15.46:0/849957112 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:03.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.515+0000 7f95ef7fe700 1 --2- 172.21.15.46:0/849957112 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f95f0100d90 0x7f95f0196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:03.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.515+0000 7f95effff700 1 -- 172.21.15.46:0/849957112 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f95f0100d90 msgr2=0x7f95f0196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:03.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.515+0000 7f95effff700 1 --2- 172.21.15.46:0/849957112 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f95f0100d90 0x7f95f0196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:03.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.515+0000 7f95effff700 1 -- 172.21.15.46:0/849957112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d80095f0 con 0x7f95f00ffb90 2024-04-27T04:02:03.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.515+0000 7f95effff700 1 --2- 172.21.15.46:0/849957112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00ffb90 0x7f95f0196670 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f95e000d900 tx=0x7f95e000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:03.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.515+0000 7f95ed7fa700 1 -- 172.21.15.46:0/849957112 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95e00041f0 con 0x7f95f00ffb90 2024-04-27T04:02:03.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.516+0000 7f95f64b8700 1 -- 172.21.15.46:0/849957112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f019bd70 con 0x7f95f00ffb90 2024-04-27T04:02:03.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.516+0000 7f95ed7fa700 1 -- 172.21.15.46:0/849957112 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95e0004350 con 0x7f95f00ffb90 2024-04-27T04:02:03.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.516+0000 7f95ed7fa700 1 -- 172.21.15.46:0/849957112 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95e0003cf0 con 0x7f95f00ffb90 2024-04-27T04:02:03.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.516+0000 7f95f64b8700 1 -- 172.21.15.46:0/849957112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f019c260 con 0x7f95f00ffb90 2024-04-27T04:02:03.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.516+0000 7f95f64b8700 1 -- 172.21.15.46:0/849957112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f004ec20 con 0x7f95f00ffb90 2024-04-27T04:02:03.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.517+0000 7f95ed7fa700 1 -- 172.21.15.46:0/849957112 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f95e0009730 con 0x7f95f00ffb90 2024-04-27T04:02:03.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.517+0000 7f95ed7fa700 1 --2- 172.21.15.46:0/849957112 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f95dc06cce0 0x7f95dc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:03.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.518+0000 7f95ed7fa700 1 -- 172.21.15.46:0/849957112 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f95e008c120 con 0x7f95f00ffb90 2024-04-27T04:02:03.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.518+0000 7f95ef7fe700 1 --2- 172.21.15.46:0/849957112 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f95dc06cce0 0x7f95dc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:03.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.518+0000 7f95ef7fe700 1 --2- 172.21.15.46:0/849957112 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f95dc06cce0 0x7f95dc06f1a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f95d8009fd0 tx=0x7f95d8005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.522+0000 7f95ed7fa700 1 -- 172.21.15.46:0/849957112 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f95f004ec20 con 0x7f95f00ffb90 2024-04-27T04:02:03.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.731+0000 7f95f64b8700 1 -- 172.21.15.46:0/849957112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f95f0108980 con 0x7f95f00ffb90 2024-04-27T04:02:03.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.735+0000 7f95ed7fa700 1 -- 172.21.15.46:0/849957112 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v25)=0 v25) v1 ==== 126+0+0 (secure 0 0 0) 0x7f95e0059f20 con 0x7f95f00ffb90 2024-04-27T04:02:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.738+0000 7f95f64b8700 1 -- 172.21.15.46:0/849957112 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f95dc06cce0 msgr2=0x7f95dc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.738+0000 7f95f64b8700 1 --2- 172.21.15.46:0/849957112 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f95dc06cce0 0x7f95dc06f1a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f95d8009fd0 tx=0x7f95d8005960 comp rx=0 tx=0).stop 2024-04-27T04:02:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.738+0000 7f95f64b8700 1 -- 172.21.15.46:0/849957112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00ffb90 msgr2=0x7f95f0196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.738+0000 7f95f64b8700 1 --2- 172.21.15.46:0/849957112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00ffb90 0x7f95f0196670 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f95e000d900 tx=0x7f95e000dc10 comp rx=0 tx=0).stop 2024-04-27T04:02:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.738+0000 7f95f64b8700 1 -- 172.21.15.46:0/849957112 shutdown_connections 2024-04-27T04:02:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.738+0000 7f95f64b8700 1 --2- 172.21.15.46:0/849957112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00ffb90 0x7f95f0196670 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.738+0000 7f95f64b8700 1 --2- 172.21.15.46:0/849957112 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f95dc06cce0 0x7f95dc06f1a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.738+0000 7f95f64b8700 1 --2- 172.21.15.46:0/849957112 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f95f0100d90 0x7f95f0196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.738+0000 7f95f64b8700 1 -- 172.21.15.46:0/849957112 >> 172.21.15.46:0/849957112 conn(0x7f95f00fb110 msgr2=0x7f95f00685c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.738+0000 7f95f64b8700 1 -- 172.21.15.46:0/849957112 shutdown_connections 2024-04-27T04:02:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:03.738+0000 7f95f64b8700 1 -- 172.21.15.46:0/849957112 wait complete. 2024-04-27T04:02:04.349 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-27T04:02:04.350 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi046.front.sepia.ceph.com 2024-04-27T04:02:04.350 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -e sha1=4e213a58e787437ff508e6c890ccd5fd32dc96b5 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-27T04:02:04.417 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:04 smithi046 ceph-mon[26899]: pgmap v200: 65 pgs: 65 active+clean; 582 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-27T04:02:04.417 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:04 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/849957112' entity='client.admin' 2024-04-27T04:02:04.417 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:04 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:02:04.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:04 smithi073 ceph-mon[34160]: pgmap v200: 65 pgs: 65 active+clean; 582 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-27T04:02:04.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:04 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/849957112' entity='client.admin' 2024-04-27T04:02:04.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:04 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:02:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:02:05.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:05 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:02:05.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:05 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:02:05.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:05 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:02:05.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:05 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:02:05.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:05 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:02:05.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:05 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:02:06.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.450+0000 7f9303f0a700 1 -- 172.21.15.46:0/1353699352 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f92fc0ffb10 msgr2=0x7f92fc0fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.450+0000 7f9303f0a700 1 --2- 172.21.15.46:0/1353699352 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f92fc0ffb10 0x7f92fc0fff30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f92ec009f90 tx=0x7f92ec00b330 comp rx=0 tx=0).stop 2024-04-27T04:02:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.450+0000 7f9303f0a700 1 -- 172.21.15.46:0/1353699352 shutdown_connections 2024-04-27T04:02:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.450+0000 7f9303f0a700 1 --2- 172.21.15.46:0/1353699352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92fc100d10 0x7f92fc101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.450+0000 7f9303f0a700 1 --2- 172.21.15.46:0/1353699352 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f92fc0ffb10 0x7f92fc0fff30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.450+0000 7f9303f0a700 1 -- 172.21.15.46:0/1353699352 >> 172.21.15.46:0/1353699352 conn(0x7f92fc0fb0b0 msgr2=0x7f92fc0fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.450+0000 7f9303f0a700 1 -- 172.21.15.46:0/1353699352 shutdown_connections 2024-04-27T04:02:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.450+0000 7f9303f0a700 1 -- 172.21.15.46:0/1353699352 wait complete. 2024-04-27T04:02:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.450+0000 7f9303f0a700 1 Processor -- start 2024-04-27T04:02:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.451+0000 7f9303f0a700 1 -- start start 2024-04-27T04:02:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.451+0000 7f9303f0a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92fc0ffb10 0x7f92fc195500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.451+0000 7f9303f0a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f92fc100d10 0x7f92fc195a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.451+0000 7f9303f0a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc196060 con 0x7f92fc0ffb10 2024-04-27T04:02:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.451+0000 7f9303f0a700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc1961a0 con 0x7f92fc100d10 2024-04-27T04:02:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.451+0000 7f9301ca6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92fc0ffb10 0x7f92fc195500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.451+0000 7f9301ca6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92fc0ffb10 0x7f92fc195500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33328/0 (socket says 172.21.15.46:33328) 2024-04-27T04:02:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.451+0000 7f9301ca6700 1 -- 172.21.15.46:0/1099419401 learned_addr learned my addr 172.21.15.46:0/1099419401 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.451+0000 7f93014a5700 1 --2- 172.21.15.46:0/1099419401 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f92fc100d10 0x7f92fc195a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.451+0000 7f9301ca6700 1 -- 172.21.15.46:0/1099419401 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f92fc100d10 msgr2=0x7f92fc195a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.451+0000 7f9301ca6700 1 --2- 172.21.15.46:0/1099419401 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f92fc100d10 0x7f92fc195a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.451+0000 7f9301ca6700 1 -- 172.21.15.46:0/1099419401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ec009c40 con 0x7f92fc0ffb10 2024-04-27T04:02:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.451+0000 7f9301ca6700 1 --2- 172.21.15.46:0/1099419401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92fc0ffb10 0x7f92fc195500 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f92ec006130 tx=0x7f92ec00bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:06.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.452+0000 7f92faffd700 1 -- 172.21.15.46:0/1099419401 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92ec00e070 con 0x7f92fc0ffb10 2024-04-27T04:02:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.452+0000 7f92faffd700 1 -- 172.21.15.46:0/1099419401 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92ec003de0 con 0x7f92fc0ffb10 2024-04-27T04:02:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.452+0000 7f9303f0a700 1 -- 172.21.15.46:0/1099419401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92fc068e50 con 0x7f92fc0ffb10 2024-04-27T04:02:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.452+0000 7f92faffd700 1 -- 172.21.15.46:0/1099419401 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92ec012620 con 0x7f92fc0ffb10 2024-04-27T04:02:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.452+0000 7f9303f0a700 1 -- 172.21.15.46:0/1099419401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92fc069420 con 0x7f92fc0ffb10 2024-04-27T04:02:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.453+0000 7f9303f0a700 1 -- 172.21.15.46:0/1099419401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92fc04ebb0 con 0x7f92fc0ffb10 2024-04-27T04:02:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.453+0000 7f92faffd700 1 -- 172.21.15.46:0/1099419401 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f92ec007ac0 con 0x7f92fc0ffb10 2024-04-27T04:02:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.454+0000 7f92faffd700 1 --2- 172.21.15.46:0/1099419401 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f92e406cce0 0x7f92e406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.454+0000 7f92faffd700 1 -- 172.21.15.46:0/1099419401 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f92ec08ed50 con 0x7f92fc0ffb10 2024-04-27T04:02:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.454+0000 7f93014a5700 1 --2- 172.21.15.46:0/1099419401 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f92e406cce0 0x7f92e406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.454+0000 7f93014a5700 1 --2- 172.21.15.46:0/1099419401 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f92e406cce0 0x7f92e406f1a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f92f0005fd0 tx=0x7f92f0009310 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.458+0000 7f92faffd700 1 -- 172.21.15.46:0/1099419401 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f92ec05cb50 con 0x7f92fc0ffb10 2024-04-27T04:02:06.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.663+0000 7f9303f0a700 1 -- 172.21.15.46:0/1099419401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f92fc108670 con 0x7f92fc0ffb10 2024-04-27T04:02:06.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.664+0000 7f92faffd700 1 -- 172.21.15.46:0/1099419401 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v25)=0 v25) v1 ==== 155+0+0 (secure 0 0 0) 0x7f92ec017030 con 0x7f92fc0ffb10 2024-04-27T04:02:06.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.666+0000 7f9303f0a700 1 -- 172.21.15.46:0/1099419401 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f92e406cce0 msgr2=0x7f92e406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:06.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.666+0000 7f9303f0a700 1 --2- 172.21.15.46:0/1099419401 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f92e406cce0 0x7f92e406f1a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f92f0005fd0 tx=0x7f92f0009310 comp rx=0 tx=0).stop 2024-04-27T04:02:06.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.666+0000 7f9303f0a700 1 -- 172.21.15.46:0/1099419401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92fc0ffb10 msgr2=0x7f92fc195500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:06.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.666+0000 7f9303f0a700 1 --2- 172.21.15.46:0/1099419401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92fc0ffb10 0x7f92fc195500 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f92ec006130 tx=0x7f92ec00bdc0 comp rx=0 tx=0).stop 2024-04-27T04:02:06.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.666+0000 7f9303f0a700 1 -- 172.21.15.46:0/1099419401 shutdown_connections 2024-04-27T04:02:06.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.666+0000 7f9303f0a700 1 --2- 172.21.15.46:0/1099419401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92fc0ffb10 0x7f92fc195500 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:06.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.666+0000 7f9303f0a700 1 --2- 172.21.15.46:0/1099419401 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f92e406cce0 0x7f92e406f1a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:06.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.666+0000 7f9303f0a700 1 --2- 172.21.15.46:0/1099419401 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f92fc100d10 0x7f92fc195a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:06.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.666+0000 7f9303f0a700 1 -- 172.21.15.46:0/1099419401 >> 172.21.15.46:0/1099419401 conn(0x7f92fc0fb0b0 msgr2=0x7f92fc104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:06.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.666+0000 7f9303f0a700 1 -- 172.21.15.46:0/1099419401 shutdown_connections 2024-04-27T04:02:06.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:06.666+0000 7f9303f0a700 1 -- 172.21.15.46:0/1099419401 wait complete. 2024-04-27T04:02:06.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:06 smithi046 ceph-mon[26899]: pgmap v201: 65 pgs: 65 active+clean; 582 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-27T04:02:07.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:06 smithi073 ceph-mon[34160]: pgmap v201: 65 pgs: 65 active+clean; 582 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-27T04:02:07.594 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -e sha1=4e213a58e787437ff508e6c890ccd5fd32dc96b5 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-27T04:02:07.858 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:02:08.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:08 smithi046 ceph-mon[26899]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-27T04:02:09.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:08 smithi073 ceph-mon[34160]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-27T04:02:09.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.487+0000 7fcdf4c69700 1 -- 172.21.15.46:0/2087918092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdf0100c30 msgr2=0x7fcdf0101050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:09.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.487+0000 7fcdf4c69700 1 --2- 172.21.15.46:0/2087918092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdf0100c30 0x7fcdf0101050 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009910 tx=0x7fcde0009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:09.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.487+0000 7fcdf4c69700 1 -- 172.21.15.46:0/2087918092 shutdown_connections 2024-04-27T04:02:09.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.487+0000 7fcdf4c69700 1 --2- 172.21.15.46:0/2087918092 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcdf0101e30 0x7fcdf0102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:09.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.487+0000 7fcdf4c69700 1 --2- 172.21.15.46:0/2087918092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdf0100c30 0x7fcdf0101050 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:09.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.487+0000 7fcdf4c69700 1 -- 172.21.15.46:0/2087918092 >> 172.21.15.46:0/2087918092 conn(0x7fcdf00fc1d0 msgr2=0x7fcdf00fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:09.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.487+0000 7fcdf4c69700 1 -- 172.21.15.46:0/2087918092 shutdown_connections 2024-04-27T04:02:09.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.487+0000 7fcdf4c69700 1 -- 172.21.15.46:0/2087918092 wait complete. 2024-04-27T04:02:09.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.488+0000 7fcdf4c69700 1 Processor -- start 2024-04-27T04:02:09.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.488+0000 7fcdf4c69700 1 -- start start 2024-04-27T04:02:09.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.489+0000 7fcdf4c69700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdf0100c30 0x7fcdf0196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:09.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.489+0000 7fcdf4c69700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcdf0101e30 0x7fcdf0196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:09.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.489+0000 7fcdf4c69700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf0197180 con 0x7fcdf0100c30 2024-04-27T04:02:09.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.489+0000 7fcdf4c69700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf01972c0 con 0x7fcdf0101e30 2024-04-27T04:02:09.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.489+0000 7fcdee59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdf0100c30 0x7fcdf0196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:09.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.489+0000 7fcdee59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdf0100c30 0x7fcdf0196620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33352/0 (socket says 172.21.15.46:33352) 2024-04-27T04:02:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.489+0000 7fcdee59c700 1 -- 172.21.15.46:0/2766859162 learned_addr learned my addr 172.21.15.46:0/2766859162 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.489+0000 7fcde5d9b700 1 --2- 172.21.15.46:0/2766859162 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcdf0101e30 0x7fcdf0196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.489+0000 7fcdee59c700 1 -- 172.21.15.46:0/2766859162 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcdf0101e30 msgr2=0x7fcdf0196b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.489+0000 7fcdee59c700 1 --2- 172.21.15.46:0/2766859162 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcdf0101e30 0x7fcdf0196b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.489+0000 7fcdee59c700 1 -- 172.21.15.46:0/2766859162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde00095f0 con 0x7fcdf0100c30 2024-04-27T04:02:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.489+0000 7fcdee59c700 1 --2- 172.21.15.46:0/2766859162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdf0100c30 0x7fcdf0196620 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcde00059d0 tx=0x7fcde000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.489+0000 7fcde7fff700 1 -- 172.21.15.46:0/2766859162 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcde001d070 con 0x7fcdf0100c30 2024-04-27T04:02:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.489+0000 7fcdf4c69700 1 -- 172.21.15.46:0/2766859162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf019bcc0 con 0x7fcdf0100c30 2024-04-27T04:02:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.489+0000 7fcde7fff700 1 -- 172.21.15.46:0/2766859162 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcde0005dc0 con 0x7fcdf0100c30 2024-04-27T04:02:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.489+0000 7fcde7fff700 1 -- 172.21.15.46:0/2766859162 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcde0021410 con 0x7fcdf0100c30 2024-04-27T04:02:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.490+0000 7fcdf4c69700 1 -- 172.21.15.46:0/2766859162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf0073110 con 0x7fcdf0100c30 2024-04-27T04:02:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.490+0000 7fcdf4c69700 1 -- 172.21.15.46:0/2766859162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf01907b0 con 0x7fcdf0100c30 2024-04-27T04:02:09.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.491+0000 7fcde7fff700 1 -- 172.21.15.46:0/2766859162 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fcde000f460 con 0x7fcdf0100c30 2024-04-27T04:02:09.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.491+0000 7fcde7fff700 1 --2- 172.21.15.46:0/2766859162 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fcddc0710f0 0x7fcddc0735b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:09.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.491+0000 7fcde7fff700 1 -- 172.21.15.46:0/2766859162 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fcde008d560 con 0x7fcdf0100c30 2024-04-27T04:02:09.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.492+0000 7fcde5d9b700 1 --2- 172.21.15.46:0/2766859162 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fcddc0710f0 0x7fcddc0735b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:09.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.492+0000 7fcde5d9b700 1 --2- 172.21.15.46:0/2766859162 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fcddc0710f0 0x7fcddc0735b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8005950 tx=0x7fcdd800b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:09.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.496+0000 7fcde7fff700 1 -- 172.21.15.46:0/2766859162 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fcde005b2c0 con 0x7fcdf0100c30 2024-04-27T04:02:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.700+0000 7fcdf4c69700 1 -- 172.21.15.46:0/2766859162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fcdf004ebb0 con 0x7fcdf0100c30 2024-04-27T04:02:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.700+0000 7fcde7fff700 1 -- 172.21.15.46:0/2766859162 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v25)=0 v25) v1 ==== 163+0+0 (secure 0 0 0) 0x7fcde005ae50 con 0x7fcdf0100c30 2024-04-27T04:02:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.702+0000 7fcdf4c69700 1 -- 172.21.15.46:0/2766859162 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fcddc0710f0 msgr2=0x7fcddc0735b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.702+0000 7fcdf4c69700 1 --2- 172.21.15.46:0/2766859162 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fcddc0710f0 0x7fcddc0735b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8005950 tx=0x7fcdd800b410 comp rx=0 tx=0).stop 2024-04-27T04:02:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.702+0000 7fcdf4c69700 1 -- 172.21.15.46:0/2766859162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdf0100c30 msgr2=0x7fcdf0196620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.702+0000 7fcdf4c69700 1 --2- 172.21.15.46:0/2766859162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdf0100c30 0x7fcdf0196620 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcde00059d0 tx=0x7fcde000bd70 comp rx=0 tx=0).stop 2024-04-27T04:02:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.702+0000 7fcdf4c69700 1 -- 172.21.15.46:0/2766859162 shutdown_connections 2024-04-27T04:02:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.702+0000 7fcdf4c69700 1 --2- 172.21.15.46:0/2766859162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdf0100c30 0x7fcdf0196620 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.702+0000 7fcdf4c69700 1 --2- 172.21.15.46:0/2766859162 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fcddc0710f0 0x7fcddc0735b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.702+0000 7fcdf4c69700 1 --2- 172.21.15.46:0/2766859162 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcdf0101e30 0x7fcdf0196b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.702+0000 7fcdf4c69700 1 -- 172.21.15.46:0/2766859162 >> 172.21.15.46:0/2766859162 conn(0x7fcdf00fc1d0 msgr2=0x7fcdf01053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.702+0000 7fcdf4c69700 1 -- 172.21.15.46:0/2766859162 shutdown_connections 2024-04-27T04:02:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:09.702+0000 7fcdf4c69700 1 -- 172.21.15.46:0/2766859162 wait complete. 2024-04-27T04:02:10.440 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -e sha1=4e213a58e787437ff508e6c890ccd5fd32dc96b5 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-27T04:02:10.659 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:02:10.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:10 smithi046 ceph-mon[26899]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-27T04:02:11.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:10 smithi073 ceph-mon[34160]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-27T04:02:12.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.351+0000 7f267a016700 1 -- 172.21.15.46:0/2641497752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2674100d20 msgr2=0x7f2674101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:12.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.351+0000 7f267a016700 1 --2- 172.21.15.46:0/2641497752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2674100d20 0x7f2674101180 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2668009910 tx=0x7f2668009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:12.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.351+0000 7f267a016700 1 -- 172.21.15.46:0/2641497752 shutdown_connections 2024-04-27T04:02:12.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.351+0000 7f267a016700 1 --2- 172.21.15.46:0/2641497752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2674100d20 0x7f2674101180 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:12.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.351+0000 7f267a016700 1 --2- 172.21.15.46:0/2641497752 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26740ffb20 0x7f26740fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:12.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.351+0000 7f267a016700 1 -- 172.21.15.46:0/2641497752 >> 172.21.15.46:0/2641497752 conn(0x7f26740fb0c0 msgr2=0x7f26740fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:12.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.351+0000 7f267a016700 1 -- 172.21.15.46:0/2641497752 shutdown_connections 2024-04-27T04:02:12.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.351+0000 7f267a016700 1 -- 172.21.15.46:0/2641497752 wait complete. 2024-04-27T04:02:12.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.351+0000 7f267a016700 1 Processor -- start 2024-04-27T04:02:12.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.351+0000 7f267a016700 1 -- start start 2024-04-27T04:02:12.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.352+0000 7f267a016700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26740ffb20 0x7f2674103a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:12.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.352+0000 7f267a016700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2674100d20 0x7f2674102100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:12.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.352+0000 7f267a016700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2674102640 con 0x7f26740ffb20 2024-04-27T04:02:12.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.352+0000 7f267a016700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2674102780 con 0x7f2674100d20 2024-04-27T04:02:12.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.352+0000 7f26737fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26740ffb20 0x7f2674103a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:12.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.352+0000 7f26737fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26740ffb20 0x7f2674103a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38378/0 (socket says 172.21.15.46:38378) 2024-04-27T04:02:12.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.352+0000 7f26737fe700 1 -- 172.21.15.46:0/3529037499 learned_addr learned my addr 172.21.15.46:0/3529037499 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:12.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.352+0000 7f2672ffd700 1 --2- 172.21.15.46:0/3529037499 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2674100d20 0x7f2674102100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:12.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.352+0000 7f26737fe700 1 -- 172.21.15.46:0/3529037499 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2674100d20 msgr2=0x7f2674102100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:12.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.352+0000 7f26737fe700 1 --2- 172.21.15.46:0/3529037499 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2674100d20 0x7f2674102100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:12.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.352+0000 7f26737fe700 1 -- 172.21.15.46:0/3529037499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26680095f0 con 0x7f26740ffb20 2024-04-27T04:02:12.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.352+0000 7f26737fe700 1 --2- 172.21.15.46:0/3529037499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26740ffb20 0x7f2674103a60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f266400b480 tx=0x7f266400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:12.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.353+0000 7f2670ff9700 1 -- 172.21.15.46:0/3529037499 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2664011800 con 0x7f26740ffb20 2024-04-27T04:02:12.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.353+0000 7f2670ff9700 1 -- 172.21.15.46:0/3529037499 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2664011e20 con 0x7f26740ffb20 2024-04-27T04:02:12.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.353+0000 7f2670ff9700 1 -- 172.21.15.46:0/3529037499 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2664010ea0 con 0x7f26740ffb20 2024-04-27T04:02:12.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.353+0000 7f267a016700 1 -- 172.21.15.46:0/3529037499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2674102a60 con 0x7f26740ffb20 2024-04-27T04:02:12.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.353+0000 7f267a016700 1 -- 172.21.15.46:0/3529037499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f267419e7d0 con 0x7f26740ffb20 2024-04-27T04:02:12.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.354+0000 7f267a016700 1 -- 172.21.15.46:0/3529037499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f267404ebb0 con 0x7f26740ffb20 2024-04-27T04:02:12.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.354+0000 7f2670ff9700 1 -- 172.21.15.46:0/3529037499 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2664010380 con 0x7f26740ffb20 2024-04-27T04:02:12.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.355+0000 7f2670ff9700 1 --2- 172.21.15.46:0/3529037499 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f265c06cce0 0x7f265c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:12.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.355+0000 7f2670ff9700 1 -- 172.21.15.46:0/3529037499 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f266408bdc0 con 0x7f26740ffb20 2024-04-27T04:02:12.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.355+0000 7f2672ffd700 1 --2- 172.21.15.46:0/3529037499 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f265c06cce0 0x7f265c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:12.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.355+0000 7f2672ffd700 1 --2- 172.21.15.46:0/3529037499 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f265c06cce0 0x7f265c06f1a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2668009fd0 tx=0x7f2668005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:12.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.360+0000 7f2670ff9700 1 -- 172.21.15.46:0/3529037499 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f2664059bc0 con 0x7f26740ffb20 2024-04-27T04:02:12.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.565+0000 7f267a016700 1 -- 172.21.15.46:0/3529037499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f2674108810 con 0x7f26740ffb20 2024-04-27T04:02:12.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.566+0000 7f2670ff9700 1 -- 172.21.15.46:0/3529037499 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v25)=0 v25) v1 ==== 135+0+0 (secure 0 0 0) 0x7f2664059750 con 0x7f26740ffb20 2024-04-27T04:02:12.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.568+0000 7f267a016700 1 -- 172.21.15.46:0/3529037499 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f265c06cce0 msgr2=0x7f265c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:12.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.568+0000 7f267a016700 1 --2- 172.21.15.46:0/3529037499 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f265c06cce0 0x7f265c06f1a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2668009fd0 tx=0x7f2668005960 comp rx=0 tx=0).stop 2024-04-27T04:02:12.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.568+0000 7f267a016700 1 -- 172.21.15.46:0/3529037499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26740ffb20 msgr2=0x7f2674103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:12.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.568+0000 7f267a016700 1 --2- 172.21.15.46:0/3529037499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26740ffb20 0x7f2674103a60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f266400b480 tx=0x7f266400b790 comp rx=0 tx=0).stop 2024-04-27T04:02:12.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.568+0000 7f267a016700 1 -- 172.21.15.46:0/3529037499 shutdown_connections 2024-04-27T04:02:12.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.568+0000 7f267a016700 1 --2- 172.21.15.46:0/3529037499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26740ffb20 0x7f2674103a60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:12.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.568+0000 7f267a016700 1 --2- 172.21.15.46:0/3529037499 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f265c06cce0 0x7f265c06f1a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:12.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.568+0000 7f267a016700 1 --2- 172.21.15.46:0/3529037499 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2674100d20 0x7f2674102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:12.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.568+0000 7f267a016700 1 -- 172.21.15.46:0/3529037499 >> 172.21.15.46:0/3529037499 conn(0x7f26740fb0c0 msgr2=0x7f2674069160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:12.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.568+0000 7f267a016700 1 -- 172.21.15.46:0/3529037499 shutdown_connections 2024-04-27T04:02:12.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:12.568+0000 7f267a016700 1 -- 172.21.15.46:0/3529037499 wait complete. 2024-04-27T04:02:12.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:12 smithi046 ceph-mon[26899]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-27T04:02:13.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:12 smithi073 ceph-mon[34160]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-27T04:02:13.306 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -e sha1=4e213a58e787437ff508e6c890ccd5fd32dc96b5 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-27T04:02:13.523 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:02:14.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:14 smithi046 ceph-mon[26899]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-27T04:02:15.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:14 smithi073 ceph-mon[34160]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-27T04:02:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.141+0000 7f07b8f8e700 1 -- 172.21.15.46:0/3968322121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07b4101780 msgr2=0x7f07b4103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.141+0000 7f07b8f8e700 1 --2- 172.21.15.46:0/3968322121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07b4101780 0x7f07b4103b70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f07a4009910 tx=0x7f07a4009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.141+0000 7f07b8f8e700 1 -- 172.21.15.46:0/3968322121 shutdown_connections 2024-04-27T04:02:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.141+0000 7f07b8f8e700 1 --2- 172.21.15.46:0/3968322121 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f07b41040b0 0x7f07b41064a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.141+0000 7f07b8f8e700 1 --2- 172.21.15.46:0/3968322121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07b4101780 0x7f07b4103b70 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.141+0000 7f07b8f8e700 1 -- 172.21.15.46:0/3968322121 >> 172.21.15.46:0/3968322121 conn(0x7f07b40fb110 msgr2=0x7f07b40fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.141+0000 7f07b8f8e700 1 -- 172.21.15.46:0/3968322121 shutdown_connections 2024-04-27T04:02:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.141+0000 7f07b8f8e700 1 -- 172.21.15.46:0/3968322121 wait complete. 2024-04-27T04:02:15.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.142+0000 7f07b8f8e700 1 Processor -- start 2024-04-27T04:02:15.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.142+0000 7f07b8f8e700 1 -- start start 2024-04-27T04:02:15.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.142+0000 7f07b8f8e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f07b4101780 0x7f07b4101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:15.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.142+0000 7f07b8f8e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07b41040b0 0x7f07b40ff730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:15.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.142+0000 7f07b8f8e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40ffc70 con 0x7f07b41040b0 2024-04-27T04:02:15.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.142+0000 7f07b8f8e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40ffdb0 con 0x7f07b4101780 2024-04-27T04:02:15.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.142+0000 7f07b1d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07b41040b0 0x7f07b40ff730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:15.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.142+0000 7f07b1d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07b41040b0 0x7f07b40ff730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38404/0 (socket says 172.21.15.46:38404) 2024-04-27T04:02:15.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.142+0000 7f07b1d9b700 1 -- 172.21.15.46:0/1599056631 learned_addr learned my addr 172.21.15.46:0/1599056631 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.142+0000 7f07b259c700 1 --2- 172.21.15.46:0/1599056631 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f07b4101780 0x7f07b4101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.143+0000 7f07b1d9b700 1 -- 172.21.15.46:0/1599056631 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f07b4101780 msgr2=0x7f07b4101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.143+0000 7f07b1d9b700 1 --2- 172.21.15.46:0/1599056631 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f07b4101780 0x7f07b4101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.143+0000 7f07b1d9b700 1 -- 172.21.15.46:0/1599056631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a40095f0 con 0x7f07b41040b0 2024-04-27T04:02:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.143+0000 7f07b1d9b700 1 --2- 172.21.15.46:0/1599056631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07b41040b0 0x7f07b40ff730 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f07a800c8a0 tx=0x7f07a800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.143+0000 7f07a37fe700 1 -- 172.21.15.46:0/1599056631 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07a80078e0 con 0x7f07b41040b0 2024-04-27T04:02:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.143+0000 7f07b8f8e700 1 -- 172.21.15.46:0/1599056631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b4100090 con 0x7f07b41040b0 2024-04-27T04:02:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.143+0000 7f07a37fe700 1 -- 172.21.15.46:0/1599056631 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07a800f450 con 0x7f07b41040b0 2024-04-27T04:02:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.143+0000 7f07a37fe700 1 -- 172.21.15.46:0/1599056631 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07a800e5e0 con 0x7f07b41040b0 2024-04-27T04:02:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.143+0000 7f07b8f8e700 1 -- 172.21.15.46:0/1599056631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b419e690 con 0x7f07b41040b0 2024-04-27T04:02:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.144+0000 7f07b8f8e700 1 -- 172.21.15.46:0/1599056631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b404ebb0 con 0x7f07b41040b0 2024-04-27T04:02:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.145+0000 7f07a37fe700 1 -- 172.21.15.46:0/1599056631 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f07a800f5c0 con 0x7f07b41040b0 2024-04-27T04:02:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.145+0000 7f07a37fe700 1 --2- 172.21.15.46:0/1599056631 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f079c06cce0 0x7f079c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.145+0000 7f07b259c700 1 --2- 172.21.15.46:0/1599056631 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f079c06cce0 0x7f079c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.145+0000 7f07a37fe700 1 -- 172.21.15.46:0/1599056631 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f07a808c320 con 0x7f07b41040b0 2024-04-27T04:02:15.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.146+0000 7f07b259c700 1 --2- 172.21.15.46:0/1599056631 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f079c06cce0 0x7f079c06f1a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f07a4005090 tx=0x7f07a4005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:15.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.150+0000 7f07a37fe700 1 -- 172.21.15.46:0/1599056631 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f07a805a080 con 0x7f07b41040b0 2024-04-27T04:02:15.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.372+0000 7f07b8f8e700 1 -- 172.21.15.46:0/1599056631 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:4e213a58e787437ff508e6c890ccd5fd32dc96b5", "target": ["mon-mgr", ""]}) v1 -- 0x7f07b4105f70 con 0x7f079c06cce0 2024-04-27T04:02:15.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.377+0000 7f07a37fe700 1 -- 172.21.15.46:0/1599056631 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f07b4105f70 con 0x7f079c06cce0 2024-04-27T04:02:15.378 INFO:teuthology.orchestra.run.smithi046.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:4e213a58e787437ff508e6c890ccd5fd32dc96b5 2024-04-27T04:02:15.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.379+0000 7f07b8f8e700 1 -- 172.21.15.46:0/1599056631 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f079c06cce0 msgr2=0x7f079c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:15.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.379+0000 7f07b8f8e700 1 --2- 172.21.15.46:0/1599056631 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f079c06cce0 0x7f079c06f1a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f07a4005090 tx=0x7f07a4005960 comp rx=0 tx=0).stop 2024-04-27T04:02:15.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.380+0000 7f07b8f8e700 1 -- 172.21.15.46:0/1599056631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07b41040b0 msgr2=0x7f07b40ff730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:15.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.380+0000 7f07b8f8e700 1 --2- 172.21.15.46:0/1599056631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07b41040b0 0x7f07b40ff730 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f07a800c8a0 tx=0x7f07a800cbb0 comp rx=0 tx=0).stop 2024-04-27T04:02:15.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.380+0000 7f07b8f8e700 1 -- 172.21.15.46:0/1599056631 shutdown_connections 2024-04-27T04:02:15.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.380+0000 7f07b8f8e700 1 --2- 172.21.15.46:0/1599056631 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f079c06cce0 0x7f079c06f1a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:15.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.380+0000 7f07b8f8e700 1 --2- 172.21.15.46:0/1599056631 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f07b4101780 0x7f07b4101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:15.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.380+0000 7f07b8f8e700 1 --2- 172.21.15.46:0/1599056631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07b41040b0 0x7f07b40ff730 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:15.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.380+0000 7f07b8f8e700 1 -- 172.21.15.46:0/1599056631 >> 172.21.15.46:0/1599056631 conn(0x7f07b40fb110 msgr2=0x7f07b40fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:15.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.380+0000 7f07b8f8e700 1 -- 172.21.15.46:0/1599056631 shutdown_connections 2024-04-27T04:02:15.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:15.380+0000 7f07b8f8e700 1 -- 172.21.15.46:0/1599056631 wait complete. 2024-04-27T04:02:16.063 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-27T04:02:16.063 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi046.front.sepia.ceph.com 2024-04-27T04:02:16.063 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -e sha1=4e213a58e787437ff508e6c890ccd5fd32dc96b5 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-27T04:02:16.283 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:02:16.633 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:16 smithi046 ceph-mon[26899]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 4 op/s 2024-04-27T04:02:16.633 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:16 smithi046 ceph-mon[26899]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:4e213a58e787437ff508e6c890ccd5fd32dc96b5", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:16.633 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:16 smithi046 ceph-mon[26899]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:4e213a58e787437ff508e6c890ccd5fd32dc96b5 2024-04-27T04:02:16.633 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:16 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:02:16.633 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:16 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:02:16.633 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:16 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:02:16.633 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:16 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:02:16.633 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:16 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:02:16.633 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:16 smithi046 ceph-mon[26899]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:4e213a58e787437ff508e6c890ccd5fd32dc96b5 2024-04-27T04:02:16.634 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:16 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:02:16.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:16 smithi073 ceph-mon[34160]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 4 op/s 2024-04-27T04:02:16.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:16 smithi073 ceph-mon[34160]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:4e213a58e787437ff508e6c890ccd5fd32dc96b5", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:16.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:16 smithi073 ceph-mon[34160]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:4e213a58e787437ff508e6c890ccd5fd32dc96b5 2024-04-27T04:02:16.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:16 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:02:16.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:16 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:02:16.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:16 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:02:16.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:16 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:02:16.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:16 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:02:16.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:16 smithi073 ceph-mon[34160]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:4e213a58e787437ff508e6c890ccd5fd32dc96b5 2024-04-27T04:02:16.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:16 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:02:18.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.037+0000 7f15c53b0700 1 -- 172.21.15.46:0/3777378448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15c0101e40 msgr2=0x7f15c01022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:18.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.037+0000 7f15c53b0700 1 --2- 172.21.15.46:0/3777378448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15c0101e40 0x7f15c01022a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009910 tx=0x7f15b4009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:18.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.037+0000 7f15c53b0700 1 -- 172.21.15.46:0/3777378448 shutdown_connections 2024-04-27T04:02:18.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.037+0000 7f15c53b0700 1 --2- 172.21.15.46:0/3777378448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15c0101e40 0x7f15c01022a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.037+0000 7f15c53b0700 1 --2- 172.21.15.46:0/3777378448 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15c0100c40 0x7f15c0101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.037+0000 7f15c53b0700 1 -- 172.21.15.46:0/3777378448 >> 172.21.15.46:0/3777378448 conn(0x7f15c00fc200 msgr2=0x7f15c00fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.037+0000 7f15c53b0700 1 -- 172.21.15.46:0/3777378448 shutdown_connections 2024-04-27T04:02:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.037+0000 7f15c53b0700 1 -- 172.21.15.46:0/3777378448 wait complete. 2024-04-27T04:02:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.038+0000 7f15c53b0700 1 Processor -- start 2024-04-27T04:02:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.038+0000 7f15c53b0700 1 -- start start 2024-04-27T04:02:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.038+0000 7f15c53b0700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15c0100c40 0x7f15c0198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.038+0000 7f15c53b0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15c0101e40 0x7f15c0198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.038+0000 7f15c53b0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c01992f0 con 0x7f15c0101e40 2024-04-27T04:02:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.038+0000 7f15c53b0700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c0199430 con 0x7f15c0100c40 2024-04-27T04:02:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.038+0000 7f15be7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15c0101e40 0x7f15c0198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.038+0000 7f15be7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15c0101e40 0x7f15c0198d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38434/0 (socket says 172.21.15.46:38434) 2024-04-27T04:02:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.039+0000 7f15be7fc700 1 -- 172.21.15.46:0/1594300282 learned_addr learned my addr 172.21.15.46:0/1594300282 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:18.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.039+0000 7f15beffd700 1 --2- 172.21.15.46:0/1594300282 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15c0100c40 0x7f15c0198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:18.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.039+0000 7f15be7fc700 1 -- 172.21.15.46:0/1594300282 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15c0100c40 msgr2=0x7f15c0198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:18.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.039+0000 7f15be7fc700 1 --2- 172.21.15.46:0/1594300282 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15c0100c40 0x7f15c0198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:18.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.039+0000 7f15be7fc700 1 -- 172.21.15.46:0/1594300282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b40095f0 con 0x7f15c0101e40 2024-04-27T04:02:18.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.039+0000 7f15beffd700 1 --2- 172.21.15.46:0/1594300282 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15c0100c40 0x7f15c0198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T04:02:18.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.039+0000 7f15be7fc700 1 --2- 172.21.15.46:0/1594300282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15c0101e40 0x7f15c0198d60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f15b4006010 tx=0x7f15b400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.039+0000 7f15a7fff700 1 -- 172.21.15.46:0/1594300282 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15b401d070 con 0x7f15c0101e40 2024-04-27T04:02:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.039+0000 7f15c53b0700 1 -- 172.21.15.46:0/1594300282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c019de40 con 0x7f15c0101e40 2024-04-27T04:02:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.039+0000 7f15c53b0700 1 -- 172.21.15.46:0/1594300282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c019e380 con 0x7f15c0101e40 2024-04-27T04:02:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.039+0000 7f15a7fff700 1 -- 172.21.15.46:0/1594300282 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15b400f460 con 0x7f15c0101e40 2024-04-27T04:02:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.039+0000 7f15a7fff700 1 -- 172.21.15.46:0/1594300282 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15b4017610 con 0x7f15c0101e40 2024-04-27T04:02:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.041+0000 7f15c53b0700 1 -- 172.21.15.46:0/1594300282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ac0052f0 con 0x7f15c0101e40 2024-04-27T04:02:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.042+0000 7f15a7fff700 1 -- 172.21.15.46:0/1594300282 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f15b4003a60 con 0x7f15c0101e40 2024-04-27T04:02:18.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.042+0000 7f15a7fff700 1 --2- 172.21.15.46:0/1594300282 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f15a806cce0 0x7f15a806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:18.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.042+0000 7f15a7fff700 1 -- 172.21.15.46:0/1594300282 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f15b408d440 con 0x7f15c0101e40 2024-04-27T04:02:18.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.042+0000 7f15beffd700 1 --2- 172.21.15.46:0/1594300282 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f15a806cce0 0x7f15a806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:18.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.043+0000 7f15beffd700 1 --2- 172.21.15.46:0/1594300282 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f15a806cce0 0x7f15a806f1a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f15b0005fd0 tx=0x7f15b0005e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:18.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.047+0000 7f15a7fff700 1 -- 172.21.15.46:0/1594300282 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f15b405b370 con 0x7f15c0101e40 2024-04-27T04:02:18.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.293+0000 7f15c53b0700 1 -- 172.21.15.46:0/1594300282 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f15ac000bc0 con 0x7f15a806cce0 2024-04-27T04:02:18.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.294+0000 7f15a7fff700 1 -- 172.21.15.46:0/1594300282 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f15ac000bc0 con 0x7f15a806cce0 2024-04-27T04:02:18.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.296+0000 7f15c53b0700 1 -- 172.21.15.46:0/1594300282 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f15a806cce0 msgr2=0x7f15a806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:18.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.296+0000 7f15c53b0700 1 --2- 172.21.15.46:0/1594300282 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f15a806cce0 0x7f15a806f1a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f15b0005fd0 tx=0x7f15b0005e40 comp rx=0 tx=0).stop 2024-04-27T04:02:18.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.296+0000 7f15c53b0700 1 -- 172.21.15.46:0/1594300282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15c0101e40 msgr2=0x7f15c0198d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:18.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.296+0000 7f15c53b0700 1 --2- 172.21.15.46:0/1594300282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15c0101e40 0x7f15c0198d60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f15b4006010 tx=0x7f15b400be20 comp rx=0 tx=0).stop 2024-04-27T04:02:18.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.296+0000 7f15c53b0700 1 -- 172.21.15.46:0/1594300282 shutdown_connections 2024-04-27T04:02:18.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.296+0000 7f15c53b0700 1 --2- 172.21.15.46:0/1594300282 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f15a806cce0 0x7f15a806f1a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:18.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.296+0000 7f15c53b0700 1 --2- 172.21.15.46:0/1594300282 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15c0100c40 0x7f15c0198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:18.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.296+0000 7f15c53b0700 1 --2- 172.21.15.46:0/1594300282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15c0101e40 0x7f15c0198d60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:18.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.296+0000 7f15c53b0700 1 -- 172.21.15.46:0/1594300282 >> 172.21.15.46:0/1594300282 conn(0x7f15c00fc200 msgr2=0x7f15c00733e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:18.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.296+0000 7f15c53b0700 1 -- 172.21.15.46:0/1594300282 shutdown_connections 2024-04-27T04:02:18.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.296+0000 7f15c53b0700 1 -- 172.21.15.46:0/1594300282 wait complete. 2024-04-27T04:02:18.307 INFO:teuthology.orchestra.run.smithi046.stdout:true 2024-04-27T04:02:18.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.411+0000 7f7f622e0700 1 -- 172.21.15.46:0/1198879307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5c100d10 msgr2=0x7f7f5c101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.411+0000 7f7f622e0700 1 --2- 172.21.15.46:0/1198879307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5c100d10 0x7f7f5c101170 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7f50009910 tx=0x7f7f50009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.411+0000 7f7f622e0700 1 -- 172.21.15.46:0/1198879307 shutdown_connections 2024-04-27T04:02:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.411+0000 7f7f622e0700 1 --2- 172.21.15.46:0/1198879307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5c100d10 0x7f7f5c101170 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.411+0000 7f7f622e0700 1 --2- 172.21.15.46:0/1198879307 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7f5c0ffb10 0x7f7f5c0fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.411+0000 7f7f622e0700 1 -- 172.21.15.46:0/1198879307 >> 172.21.15.46:0/1198879307 conn(0x7f7f5c0fb0b0 msgr2=0x7f7f5c0fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.411+0000 7f7f622e0700 1 -- 172.21.15.46:0/1198879307 shutdown_connections 2024-04-27T04:02:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.411+0000 7f7f622e0700 1 -- 172.21.15.46:0/1198879307 wait complete. 2024-04-27T04:02:18.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.412+0000 7f7f622e0700 1 Processor -- start 2024-04-27T04:02:18.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.412+0000 7f7f622e0700 1 -- start start 2024-04-27T04:02:18.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.412+0000 7f7f622e0700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7f5c0ffb10 0x7f7f5c196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:18.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.412+0000 7f7f622e0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5c100d10 0x7f7f5c196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:18.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.412+0000 7f7f622e0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c1971a0 con 0x7f7f5c100d10 2024-04-27T04:02:18.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.412+0000 7f7f622e0700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c1972e0 con 0x7f7f5c0ffb10 2024-04-27T04:02:18.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.412+0000 7f7f5b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5c100d10 0x7f7f5c196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:18.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.412+0000 7f7f5b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5c100d10 0x7f7f5c196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38458/0 (socket says 172.21.15.46:38458) 2024-04-27T04:02:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.412+0000 7f7f5b7fe700 1 -- 172.21.15.46:0/3875623655 learned_addr learned my addr 172.21.15.46:0/3875623655 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.412+0000 7f7f5bfff700 1 --2- 172.21.15.46:0/3875623655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7f5c0ffb10 0x7f7f5c196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.412+0000 7f7f5b7fe700 1 -- 172.21.15.46:0/3875623655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7f5c0ffb10 msgr2=0x7f7f5c196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.413+0000 7f7f5b7fe700 1 --2- 172.21.15.46:0/3875623655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7f5c0ffb10 0x7f7f5c196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.413+0000 7f7f5b7fe700 1 -- 172.21.15.46:0/3875623655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f500095f0 con 0x7f7f5c100d10 2024-04-27T04:02:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.413+0000 7f7f5bfff700 1 --2- 172.21.15.46:0/3875623655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7f5c0ffb10 0x7f7f5c196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T04:02:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.413+0000 7f7f5b7fe700 1 --2- 172.21.15.46:0/3875623655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5c100d10 0x7f7f5c196b80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7f500092e0 tx=0x7f7f5000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.413+0000 7f7f597fa700 1 -- 172.21.15.46:0/3875623655 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f5001d070 con 0x7f7f5c100d10 2024-04-27T04:02:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.413+0000 7f7f622e0700 1 -- 172.21.15.46:0/3875623655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5c19bc90 con 0x7f7f5c100d10 2024-04-27T04:02:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.413+0000 7f7f622e0700 1 -- 172.21.15.46:0/3875623655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f5c19c200 con 0x7f7f5c100d10 2024-04-27T04:02:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.413+0000 7f7f597fa700 1 -- 172.21.15.46:0/3875623655 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f5000f460 con 0x7f7f5c100d10 2024-04-27T04:02:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.413+0000 7f7f597fa700 1 -- 172.21.15.46:0/3875623655 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f50021410 con 0x7f7f5c100d10 2024-04-27T04:02:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.414+0000 7f7f622e0700 1 -- 172.21.15.46:0/3875623655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f480052f0 con 0x7f7f5c100d10 2024-04-27T04:02:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.415+0000 7f7f597fa700 1 -- 172.21.15.46:0/3875623655 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7f5000f5d0 con 0x7f7f5c100d10 2024-04-27T04:02:18.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.415+0000 7f7f597fa700 1 --2- 172.21.15.46:0/3875623655 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7f4406cce0 0x7f7f4406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:18.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.415+0000 7f7f597fa700 1 -- 172.21.15.46:0/3875623655 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7f5008d490 con 0x7f7f5c100d10 2024-04-27T04:02:18.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.416+0000 7f7f5bfff700 1 --2- 172.21.15.46:0/3875623655 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7f4406cce0 0x7f7f4406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:18.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.416+0000 7f7f5bfff700 1 --2- 172.21.15.46:0/3875623655 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7f4406cce0 0x7f7f4406f1a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c005950 tx=0x7f7f4c0058e0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:18.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.420+0000 7f7f597fa700 1 -- 172.21.15.46:0/3875623655 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f7f5005b290 con 0x7f7f5c100d10 2024-04-27T04:02:18.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:18 smithi046 ceph-mon[26899]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 4 op/s 2024-04-27T04:02:18.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.681+0000 7f7f622e0700 1 -- 172.21.15.46:0/3875623655 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f48000bc0 con 0x7f7f4406cce0 2024-04-27T04:02:18.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.682+0000 7f7f597fa700 1 -- 172.21.15.46:0/3875623655 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f7f48000bc0 con 0x7f7f4406cce0 2024-04-27T04:02:18.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.684+0000 7f7f622e0700 1 -- 172.21.15.46:0/3875623655 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7f4406cce0 msgr2=0x7f7f4406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:18.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.684+0000 7f7f622e0700 1 --2- 172.21.15.46:0/3875623655 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7f4406cce0 0x7f7f4406f1a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c005950 tx=0x7f7f4c0058e0 comp rx=0 tx=0).stop 2024-04-27T04:02:18.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.684+0000 7f7f622e0700 1 -- 172.21.15.46:0/3875623655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5c100d10 msgr2=0x7f7f5c196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:18.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.684+0000 7f7f622e0700 1 --2- 172.21.15.46:0/3875623655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5c100d10 0x7f7f5c196b80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7f500092e0 tx=0x7f7f5000be20 comp rx=0 tx=0).stop 2024-04-27T04:02:18.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.684+0000 7f7f622e0700 1 -- 172.21.15.46:0/3875623655 shutdown_connections 2024-04-27T04:02:18.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.684+0000 7f7f622e0700 1 --2- 172.21.15.46:0/3875623655 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7f4406cce0 0x7f7f4406f1a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:18.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.684+0000 7f7f622e0700 1 --2- 172.21.15.46:0/3875623655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7f5c0ffb10 0x7f7f5c196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:18.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.684+0000 7f7f622e0700 1 --2- 172.21.15.46:0/3875623655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5c100d10 0x7f7f5c196b80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:18.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.684+0000 7f7f622e0700 1 -- 172.21.15.46:0/3875623655 >> 172.21.15.46:0/3875623655 conn(0x7f7f5c0fb0b0 msgr2=0x7f7f5c068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:18.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.684+0000 7f7f622e0700 1 -- 172.21.15.46:0/3875623655 shutdown_connections 2024-04-27T04:02:18.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.685+0000 7f7f622e0700 1 -- 172.21.15.46:0/3875623655 wait complete. 2024-04-27T04:02:18.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.811+0000 7f1f5f73d700 1 -- 172.21.15.46:0/2266593386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f58101720 msgr2=0x7f1f58103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:18.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.811+0000 7f1f5f73d700 1 --2- 172.21.15.46:0/2266593386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f58101720 0x7f1f58103b10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1f54009910 tx=0x7f1f54009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:18.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.811+0000 7f1f5f73d700 1 -- 172.21.15.46:0/2266593386 shutdown_connections 2024-04-27T04:02:18.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.811+0000 7f1f5f73d700 1 --2- 172.21.15.46:0/2266593386 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f58104050 0x7f1f58106440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:18.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.811+0000 7f1f5f73d700 1 --2- 172.21.15.46:0/2266593386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f58101720 0x7f1f58103b10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:18.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.811+0000 7f1f5f73d700 1 -- 172.21.15.46:0/2266593386 >> 172.21.15.46:0/2266593386 conn(0x7f1f580fb110 msgr2=0x7f1f580fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:18.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.811+0000 7f1f5f73d700 1 -- 172.21.15.46:0/2266593386 shutdown_connections 2024-04-27T04:02:18.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.811+0000 7f1f5f73d700 1 -- 172.21.15.46:0/2266593386 wait complete. 2024-04-27T04:02:18.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.811+0000 7f1f5f73d700 1 Processor -- start 2024-04-27T04:02:18.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.812+0000 7f1f5f73d700 1 -- start start 2024-04-27T04:02:18.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.812+0000 7f1f5f73d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f58101720 0x7f1f58196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:18.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.812+0000 7f1f5f73d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f58104050 0x7f1f58196bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:18.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.812+0000 7f1f5f73d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f581971e0 con 0x7f1f58101720 2024-04-27T04:02:18.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.812+0000 7f1f5f73d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f58197320 con 0x7f1f58104050 2024-04-27T04:02:18.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.812+0000 7f1f5d4d9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f58101720 0x7f1f58196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:18.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.812+0000 7f1f5d4d9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f58101720 0x7f1f58196680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38478/0 (socket says 172.21.15.46:38478) 2024-04-27T04:02:18.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.812+0000 7f1f5d4d9700 1 -- 172.21.15.46:0/2582243166 learned_addr learned my addr 172.21.15.46:0/2582243166 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:18.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.812+0000 7f1f5ccd8700 1 --2- 172.21.15.46:0/2582243166 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f58104050 0x7f1f58196bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:18.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.812+0000 7f1f5d4d9700 1 -- 172.21.15.46:0/2582243166 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f58104050 msgr2=0x7f1f58196bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:18.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.812+0000 7f1f5d4d9700 1 --2- 172.21.15.46:0/2582243166 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f58104050 0x7f1f58196bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:18.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.812+0000 7f1f5d4d9700 1 -- 172.21.15.46:0/2582243166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f540095f0 con 0x7f1f58101720 2024-04-27T04:02:18.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.812+0000 7f1f5d4d9700 1 --2- 172.21.15.46:0/2582243166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f58101720 0x7f1f58196680 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1f54006010 tx=0x7f1f5400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:18.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.813+0000 7f1f4e7fc700 1 -- 172.21.15.46:0/2582243166 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f5401d070 con 0x7f1f58101720 2024-04-27T04:02:18.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.813+0000 7f1f5f73d700 1 -- 172.21.15.46:0/2582243166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f5819bd20 con 0x7f1f58101720 2024-04-27T04:02:18.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.813+0000 7f1f4e7fc700 1 -- 172.21.15.46:0/2582243166 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f5400f460 con 0x7f1f58101720 2024-04-27T04:02:18.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.813+0000 7f1f4e7fc700 1 -- 172.21.15.46:0/2582243166 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f54021620 con 0x7f1f58101720 2024-04-27T04:02:18.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.813+0000 7f1f5f73d700 1 -- 172.21.15.46:0/2582243166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f5819c240 con 0x7f1f58101720 2024-04-27T04:02:18.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.814+0000 7f1f5f73d700 1 -- 172.21.15.46:0/2582243166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f5804ebb0 con 0x7f1f58101720 2024-04-27T04:02:18.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.814+0000 7f1f4e7fc700 1 -- 172.21.15.46:0/2582243166 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1f5400f5d0 con 0x7f1f58101720 2024-04-27T04:02:18.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.815+0000 7f1f4e7fc700 1 --2- 172.21.15.46:0/2582243166 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1f4406cce0 0x7f1f4406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:18.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.815+0000 7f1f4e7fc700 1 -- 172.21.15.46:0/2582243166 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1f5408d4e0 con 0x7f1f58101720 2024-04-27T04:02:18.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.815+0000 7f1f5ccd8700 1 --2- 172.21.15.46:0/2582243166 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1f4406cce0 0x7f1f4406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:18.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.815+0000 7f1f5ccd8700 1 --2- 172.21.15.46:0/2582243166 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1f4406cce0 0x7f1f4406f1a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1f48009a10 tx=0x7f1f48009190 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:18.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:18.820+0000 7f1f4e7fc700 1 -- 172.21.15.46:0/2582243166 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f1f5405b360 con 0x7f1f58101720 2024-04-27T04:02:18.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:18 smithi073 ceph-mon[34160]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 4 op/s 2024-04-27T04:02:19.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.071+0000 7f1f5f73d700 1 -- 172.21.15.46:0/2582243166 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f58105ee0 con 0x7f1f4406cce0 2024-04-27T04:02:19.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.082+0000 7f1f4e7fc700 1 -- 172.21.15.46:0/2582243166 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f1f58105ee0 con 0x7f1f4406cce0 2024-04-27T04:02:19.083 INFO:teuthology.orchestra.run.smithi046.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T04:02:19.083 INFO:teuthology.orchestra.run.smithi046.stdout:alertmanager.smithi046 smithi046 *:9093,9094 running (4m) 27s ago 6m 24.8M - 0.25.0 c8568f914cd2 698cffee765e 2024-04-27T04:02:19.083 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-exporter.smithi046 smithi046 running (6m) 27s ago 6m 7570k - 18.2.2-1260-g5403a021 267a473d505d 6eec9c927d3f 2024-04-27T04:02:19.083 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-exporter.smithi073 smithi073 running (5m) 32s ago 5m 7478k - 18.2.2-1260-g5403a021 267a473d505d d101eef4888d 2024-04-27T04:02:19.083 INFO:teuthology.orchestra.run.smithi046.stdout:crash.smithi046 smithi046 running (6m) 27s ago 6m 6660k - 18.2.2-1260-g5403a021 267a473d505d 338c449faabd 2024-04-27T04:02:19.083 INFO:teuthology.orchestra.run.smithi046.stdout:crash.smithi073 smithi073 running (5m) 32s ago 5m 6660k - 18.2.2-1260-g5403a021 267a473d505d be1aaee1c655 2024-04-27T04:02:19.083 INFO:teuthology.orchestra.run.smithi046.stdout:grafana.smithi046 smithi046 *:3000 running (4m) 27s ago 6m 84.6M - 9.4.7 954c08fa6188 a834a3988304 2024-04-27T04:02:19.083 INFO:teuthology.orchestra.run.smithi046.stdout:mds.cephfs.smithi046.mycakd smithi046 running (46s) 27s ago 45s 16.4M - 18.2.2-1260-g5403a021 267a473d505d 74d9a0b351ac 2024-04-27T04:02:19.084 INFO:teuthology.orchestra.run.smithi046.stdout:mds.cephfs.smithi046.qgmaql smithi046 running (53s) 27s ago 53s 12.9M - 18.2.2-1260-g5403a021 267a473d505d 741df55feeaf 2024-04-27T04:02:19.084 INFO:teuthology.orchestra.run.smithi046.stdout:mds.cephfs.smithi073.yryono smithi073 running (43s) 32s ago 43s 16.5M - 18.2.2-1260-g5403a021 267a473d505d 40d61213120b 2024-04-27T04:02:19.084 INFO:teuthology.orchestra.run.smithi046.stdout:mds.cephfs.smithi073.zkoypj smithi073 running (50s) 32s ago 50s 18.5M - 18.2.2-1260-g5403a021 267a473d505d 5812a431fb59 2024-04-27T04:02:19.084 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.smithi046.qrfdvr smithi046 *:9283,8765,8443 running (8m) 27s ago 8m 498M - 18.2.2-1260-g5403a021 267a473d505d ba1d99ba4a26 2024-04-27T04:02:19.084 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.smithi073.awyllt smithi073 *:8443,9283,8765 running (5m) 32s ago 5m 451M - 18.2.2-1260-g5403a021 267a473d505d 06769682a51f 2024-04-27T04:02:19.084 INFO:teuthology.orchestra.run.smithi046.stdout:mon.smithi046 smithi046 running (8m) 27s ago 8m 44.1M 2048M 18.2.2-1260-g5403a021 267a473d505d 72fac097effc 2024-04-27T04:02:19.084 INFO:teuthology.orchestra.run.smithi046.stdout:mon.smithi073 smithi073 running (5m) 32s ago 5m 35.2M 2048M 18.2.2-1260-g5403a021 267a473d505d 77769d199765 2024-04-27T04:02:19.084 INFO:teuthology.orchestra.run.smithi046.stdout:node-exporter.smithi046 smithi046 *:9100 running (6m) 27s ago 6m 16.3M - 1.5.0 0da6a335fe13 02c01746eff1 2024-04-27T04:02:19.084 INFO:teuthology.orchestra.run.smithi046.stdout:node-exporter.smithi073 smithi073 *:9100 running (5m) 32s ago 5m 15.2M - 1.5.0 0da6a335fe13 377ef4267056 2024-04-27T04:02:19.084 INFO:teuthology.orchestra.run.smithi046.stdout:osd.0 smithi046 running (4m) 27s ago 4m 57.4M 3948M 18.2.2-1260-g5403a021 267a473d505d 563c995c0df6 2024-04-27T04:02:19.084 INFO:teuthology.orchestra.run.smithi046.stdout:osd.1 smithi046 running (4m) 27s ago 4m 48.7M 3948M 18.2.2-1260-g5403a021 267a473d505d 5673c75e53bb 2024-04-27T04:02:19.084 INFO:teuthology.orchestra.run.smithi046.stdout:osd.2 smithi046 running (3m) 27s ago 3m 46.7M 3948M 18.2.2-1260-g5403a021 267a473d505d 9787b5012842 2024-04-27T04:02:19.085 INFO:teuthology.orchestra.run.smithi046.stdout:osd.3 smithi073 running (3m) 32s ago 3m 46.3M 4970M 18.2.2-1260-g5403a021 267a473d505d 0c89b358284d 2024-04-27T04:02:19.085 INFO:teuthology.orchestra.run.smithi046.stdout:osd.4 smithi073 running (2m) 32s ago 2m 53.4M 4970M 18.2.2-1260-g5403a021 267a473d505d 26f06c87c31e 2024-04-27T04:02:19.085 INFO:teuthology.orchestra.run.smithi046.stdout:osd.5 smithi073 running (2m) 32s ago 2m 49.6M 4970M 18.2.2-1260-g5403a021 267a473d505d e1ce06e0da9b 2024-04-27T04:02:19.085 INFO:teuthology.orchestra.run.smithi046.stdout:prometheus.smithi046 smithi046 *:9095 running (4m) 27s ago 5m 66.2M - 2.43.0 a07b618ecd1d e1a7d8656893 2024-04-27T04:02:19.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.084+0000 7f1f5f73d700 1 -- 172.21.15.46:0/2582243166 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1f4406cce0 msgr2=0x7f1f4406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:19.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.084+0000 7f1f5f73d700 1 --2- 172.21.15.46:0/2582243166 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1f4406cce0 0x7f1f4406f1a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1f48009a10 tx=0x7f1f48009190 comp rx=0 tx=0).stop 2024-04-27T04:02:19.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.084+0000 7f1f5f73d700 1 -- 172.21.15.46:0/2582243166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f58101720 msgr2=0x7f1f58196680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:19.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.084+0000 7f1f5f73d700 1 --2- 172.21.15.46:0/2582243166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f58101720 0x7f1f58196680 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1f54006010 tx=0x7f1f5400bd70 comp rx=0 tx=0).stop 2024-04-27T04:02:19.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.085+0000 7f1f5f73d700 1 -- 172.21.15.46:0/2582243166 shutdown_connections 2024-04-27T04:02:19.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.085+0000 7f1f5f73d700 1 --2- 172.21.15.46:0/2582243166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f58101720 0x7f1f58196680 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:19.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.085+0000 7f1f5f73d700 1 --2- 172.21.15.46:0/2582243166 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f1f4406cce0 0x7f1f4406f1a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:19.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.085+0000 7f1f5f73d700 1 --2- 172.21.15.46:0/2582243166 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f58104050 0x7f1f58196bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:19.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.085+0000 7f1f5f73d700 1 -- 172.21.15.46:0/2582243166 >> 172.21.15.46:0/2582243166 conn(0x7f1f580fb110 msgr2=0x7f1f580fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:19.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.085+0000 7f1f5f73d700 1 -- 172.21.15.46:0/2582243166 shutdown_connections 2024-04-27T04:02:19.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.085+0000 7f1f5f73d700 1 -- 172.21.15.46:0/2582243166 wait complete. 2024-04-27T04:02:19.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.196+0000 7f869e6b7700 1 -- 172.21.15.46:0/1047520859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8698100d90 msgr2=0x7f86981011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:19.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.196+0000 7f869e6b7700 1 --2- 172.21.15.46:0/1047520859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8698100d90 0x7f86981011f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f868c009910 tx=0x7f868c009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:19.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.197+0000 7f869e6b7700 1 -- 172.21.15.46:0/1047520859 shutdown_connections 2024-04-27T04:02:19.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.197+0000 7f869e6b7700 1 --2- 172.21.15.46:0/1047520859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8698100d90 0x7f86981011f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:19.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.197+0000 7f869e6b7700 1 --2- 172.21.15.46:0/1047520859 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f86980ffb90 0x7f86980fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:19.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.197+0000 7f869e6b7700 1 -- 172.21.15.46:0/1047520859 >> 172.21.15.46:0/1047520859 conn(0x7f86980fb110 msgr2=0x7f86980fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:19.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.197+0000 7f869e6b7700 1 -- 172.21.15.46:0/1047520859 shutdown_connections 2024-04-27T04:02:19.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.197+0000 7f869e6b7700 1 -- 172.21.15.46:0/1047520859 wait complete. 2024-04-27T04:02:19.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.197+0000 7f869e6b7700 1 Processor -- start 2024-04-27T04:02:19.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.197+0000 7f869e6b7700 1 -- start start 2024-04-27T04:02:19.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.198+0000 7f869e6b7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f86980ffb90 0x7f8698198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:19.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.198+0000 7f869e6b7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8698100d90 0x7f8698198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:19.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.198+0000 7f869e6b7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86981993f0 con 0x7f8698100d90 2024-04-27T04:02:19.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.198+0000 7f869e6b7700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8698199530 con 0x7f86980ffb90 2024-04-27T04:02:19.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.198+0000 7f86977fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8698100d90 0x7f8698198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:19.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.198+0000 7f86977fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8698100d90 0x7f8698198dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38492/0 (socket says 172.21.15.46:38492) 2024-04-27T04:02:19.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.198+0000 7f86977fe700 1 -- 172.21.15.46:0/2990771200 learned_addr learned my addr 172.21.15.46:0/2990771200 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:19.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.198+0000 7f8697fff700 1 --2- 172.21.15.46:0/2990771200 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f86980ffb90 0x7f8698198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:19.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.198+0000 7f86977fe700 1 -- 172.21.15.46:0/2990771200 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f86980ffb90 msgr2=0x7f8698198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:19.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.198+0000 7f86977fe700 1 --2- 172.21.15.46:0/2990771200 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f86980ffb90 0x7f8698198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:19.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.198+0000 7f86977fe700 1 -- 172.21.15.46:0/2990771200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868c0095f0 con 0x7f8698100d90 2024-04-27T04:02:19.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.198+0000 7f86977fe700 1 --2- 172.21.15.46:0/2990771200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8698100d90 0x7f8698198dd0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f868c0092e0 tx=0x7f868c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:19.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.199+0000 7f86957fa700 1 -- 172.21.15.46:0/2990771200 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f868c01d070 con 0x7f8698100d90 2024-04-27T04:02:19.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.199+0000 7f86957fa700 1 -- 172.21.15.46:0/2990771200 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f868c00f460 con 0x7f8698100d90 2024-04-27T04:02:19.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.199+0000 7f869e6b7700 1 -- 172.21.15.46:0/2990771200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869819df30 con 0x7f8698100d90 2024-04-27T04:02:19.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.199+0000 7f86957fa700 1 -- 172.21.15.46:0/2990771200 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f868c021620 con 0x7f8698100d90 2024-04-27T04:02:19.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.199+0000 7f869e6b7700 1 -- 172.21.15.46:0/2990771200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869819e3f0 con 0x7f8698100d90 2024-04-27T04:02:19.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.200+0000 7f869e6b7700 1 -- 172.21.15.46:0/2990771200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869804ebb0 con 0x7f8698100d90 2024-04-27T04:02:19.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.200+0000 7f86957fa700 1 -- 172.21.15.46:0/2990771200 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f868c00f5d0 con 0x7f8698100d90 2024-04-27T04:02:19.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.201+0000 7f86957fa700 1 --2- 172.21.15.46:0/2990771200 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f868006cce0 0x7f868006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:19.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.201+0000 7f86957fa700 1 -- 172.21.15.46:0/2990771200 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f868c08d4c0 con 0x7f8698100d90 2024-04-27T04:02:19.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.201+0000 7f8697fff700 1 --2- 172.21.15.46:0/2990771200 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f868006cce0 0x7f868006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:19.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.201+0000 7f8697fff700 1 --2- 172.21.15.46:0/2990771200 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f868006cce0 0x7f868006f1a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8698100bf0 tx=0x7f868800b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:19.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.206+0000 7f86957fa700 1 -- 172.21.15.46:0/2990771200 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f868c05b2c0 con 0x7f8698100d90 2024-04-27T04:02:19.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.527+0000 7f869e6b7700 1 -- 172.21.15.46:0/2990771200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f86981086f0 con 0x7f8698100d90 2024-04-27T04:02:19.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.527+0000 7f86957fa700 1 -- 172.21.15.46:0/2990771200 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f868c05ae50 con 0x7f8698100d90 2024-04-27T04:02:19.528 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-27T04:02:19.528 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": { 2024-04-27T04:02:19.528 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)": 2 2024-04-27T04:02:19.528 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-27T04:02:19.528 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": { 2024-04-27T04:02:19.528 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)": 2 2024-04-27T04:02:19.529 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-27T04:02:19.529 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": { 2024-04-27T04:02:19.529 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)": 6 2024-04-27T04:02:19.529 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-27T04:02:19.529 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": { 2024-04-27T04:02:19.529 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)": 4 2024-04-27T04:02:19.529 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-27T04:02:19.529 INFO:teuthology.orchestra.run.smithi046.stdout: "overall": { 2024-04-27T04:02:19.529 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)": 14 2024-04-27T04:02:19.529 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-27T04:02:19.529 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-27T04:02:19.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.530+0000 7f869e6b7700 1 -- 172.21.15.46:0/2990771200 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f868006cce0 msgr2=0x7f868006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:19.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.530+0000 7f869e6b7700 1 --2- 172.21.15.46:0/2990771200 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f868006cce0 0x7f868006f1a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8698100bf0 tx=0x7f868800b410 comp rx=0 tx=0).stop 2024-04-27T04:02:19.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.530+0000 7f869e6b7700 1 -- 172.21.15.46:0/2990771200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8698100d90 msgr2=0x7f8698198dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:19.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.530+0000 7f869e6b7700 1 --2- 172.21.15.46:0/2990771200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8698100d90 0x7f8698198dd0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f868c0092e0 tx=0x7f868c00bd70 comp rx=0 tx=0).stop 2024-04-27T04:02:19.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.530+0000 7f869e6b7700 1 -- 172.21.15.46:0/2990771200 shutdown_connections 2024-04-27T04:02:19.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.530+0000 7f869e6b7700 1 --2- 172.21.15.46:0/2990771200 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f868006cce0 0x7f868006f1a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:19.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.530+0000 7f869e6b7700 1 --2- 172.21.15.46:0/2990771200 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f86980ffb90 0x7f8698198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:19.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.530+0000 7f869e6b7700 1 --2- 172.21.15.46:0/2990771200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8698100d90 0x7f8698198dd0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:19.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.530+0000 7f869e6b7700 1 -- 172.21.15.46:0/2990771200 >> 172.21.15.46:0/2990771200 conn(0x7f86980fb110 msgr2=0x7f86980685f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:19.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.530+0000 7f869e6b7700 1 -- 172.21.15.46:0/2990771200 shutdown_connections 2024-04-27T04:02:19.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.530+0000 7f869e6b7700 1 -- 172.21.15.46:0/2990771200 wait complete. 2024-04-27T04:02:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.646+0000 7fab3a55d700 1 -- 172.21.15.46:0/3758367478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab340ffb10 msgr2=0x7fab340fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.646+0000 7fab3a55d700 1 --2- 172.21.15.46:0/3758367478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab340ffb10 0x7fab340fff30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fab24009910 tx=0x7fab24009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.647+0000 7fab3a55d700 1 -- 172.21.15.46:0/3758367478 shutdown_connections 2024-04-27T04:02:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.647+0000 7fab3a55d700 1 --2- 172.21.15.46:0/3758367478 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab34100d10 0x7fab34101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.647+0000 7fab3a55d700 1 --2- 172.21.15.46:0/3758367478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab340ffb10 0x7fab340fff30 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.647+0000 7fab3a55d700 1 -- 172.21.15.46:0/3758367478 >> 172.21.15.46:0/3758367478 conn(0x7fab340fb0b0 msgr2=0x7fab340fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:19.648 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:19 smithi046 ceph-mon[26899]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:19.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.647+0000 7fab3a55d700 1 -- 172.21.15.46:0/3758367478 shutdown_connections 2024-04-27T04:02:19.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.647+0000 7fab3a55d700 1 -- 172.21.15.46:0/3758367478 wait complete. 2024-04-27T04:02:19.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.648+0000 7fab3a55d700 1 Processor -- start 2024-04-27T04:02:19.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.648+0000 7fab3a55d700 1 -- start start 2024-04-27T04:02:19.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.648+0000 7fab3a55d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab340ffb10 0x7fab34069960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:19.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.648+0000 7fab3a55d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab34100d10 0x7fab3419b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:19.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.648+0000 7fab3a55d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3419b8d0 con 0x7fab340ffb10 2024-04-27T04:02:19.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.648+0000 7fab3a55d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3419ba40 con 0x7fab34100d10 2024-04-27T04:02:19.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.648+0000 7fab33fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab340ffb10 0x7fab34069960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:19.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.648+0000 7fab33fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab340ffb10 0x7fab34069960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38514/0 (socket says 172.21.15.46:38514) 2024-04-27T04:02:19.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.648+0000 7fab33fff700 1 -- 172.21.15.46:0/3660237743 learned_addr learned my addr 172.21.15.46:0/3660237743 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:19.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.649+0000 7fab337fe700 1 --2- 172.21.15.46:0/3660237743 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab34100d10 0x7fab3419b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:19.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.649+0000 7fab33fff700 1 -- 172.21.15.46:0/3660237743 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab34100d10 msgr2=0x7fab3419b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:19.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.649+0000 7fab33fff700 1 --2- 172.21.15.46:0/3660237743 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab34100d10 0x7fab3419b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:19.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.649+0000 7fab33fff700 1 -- 172.21.15.46:0/3660237743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab240095f0 con 0x7fab340ffb10 2024-04-27T04:02:19.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.649+0000 7fab33fff700 1 --2- 172.21.15.46:0/3660237743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab340ffb10 0x7fab34069960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fab2400bd70 tx=0x7fab2400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:19.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.649+0000 7fab317fa700 1 -- 172.21.15.46:0/3660237743 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab2401d070 con 0x7fab340ffb10 2024-04-27T04:02:19.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.649+0000 7fab317fa700 1 -- 172.21.15.46:0/3660237743 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab2400f460 con 0x7fab340ffb10 2024-04-27T04:02:19.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.649+0000 7fab3a55d700 1 -- 172.21.15.46:0/3660237743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3419bcc0 con 0x7fab340ffb10 2024-04-27T04:02:19.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.649+0000 7fab317fa700 1 -- 172.21.15.46:0/3660237743 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab24021600 con 0x7fab340ffb10 2024-04-27T04:02:19.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.650+0000 7fab3a55d700 1 -- 172.21.15.46:0/3660237743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3419c1b0 con 0x7fab340ffb10 2024-04-27T04:02:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.651+0000 7fab317fa700 1 -- 172.21.15.46:0/3660237743 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fab24003a60 con 0x7fab340ffb10 2024-04-27T04:02:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.651+0000 7fab1affd700 1 -- 172.21.15.46:0/3660237743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3404ebb0 con 0x7fab340ffb10 2024-04-27T04:02:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.652+0000 7fab317fa700 1 --2- 172.21.15.46:0/3660237743 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fab1c06ccf0 0x7fab1c06f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.652+0000 7fab317fa700 1 -- 172.21.15.46:0/3660237743 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fab2408d520 con 0x7fab340ffb10 2024-04-27T04:02:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.652+0000 7fab337fe700 1 --2- 172.21.15.46:0/3660237743 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fab1c06ccf0 0x7fab1c06f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.652+0000 7fab337fe700 1 --2- 172.21.15.46:0/3660237743 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fab1c06ccf0 0x7fab1c06f1b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fab28005950 tx=0x7fab28005cf0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.657+0000 7fab317fa700 1 -- 172.21.15.46:0/3660237743 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fab2405b3a0 con 0x7fab340ffb10 2024-04-27T04:02:19.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:19 smithi073 ceph-mon[34160]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:19.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.939+0000 7fab1affd700 1 -- 172.21.15.46:0/3660237743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fab34108670 con 0x7fab340ffb10 2024-04-27T04:02:19.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.939+0000 7fab317fa700 1 -- 172.21.15.46:0/3660237743 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1865 (secure 0 0 0) 0x7fab2405af30 con 0x7fab340ffb10 2024-04-27T04:02:19.941 INFO:teuthology.orchestra.run.smithi046.stdout:e19 2024-04-27T04:02:19.941 INFO:teuthology.orchestra.run.smithi046.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T04:02:19.941 INFO:teuthology.orchestra.run.smithi046.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T04:02:19.942 INFO:teuthology.orchestra.run.smithi046.stdout:legacy client fscid: 1 2024-04-27T04:02:19.942 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:02:19.942 INFO:teuthology.orchestra.run.smithi046.stdout:Filesystem 'cephfs' (1) 2024-04-27T04:02:19.942 INFO:teuthology.orchestra.run.smithi046.stdout:fs_name cephfs 2024-04-27T04:02:19.942 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 19 2024-04-27T04:02:19.943 INFO:teuthology.orchestra.run.smithi046.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-27T04:02:19.943 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-04-27T04:01:21.676811+0000 2024-04-27T04:02:19.943 INFO:teuthology.orchestra.run.smithi046.stdout:modified 2024-04-27T04:01:42.132288+0000 2024-04-27T04:02:19.943 INFO:teuthology.orchestra.run.smithi046.stdout:tableserver 0 2024-04-27T04:02:19.943 INFO:teuthology.orchestra.run.smithi046.stdout:root 0 2024-04-27T04:02:19.944 INFO:teuthology.orchestra.run.smithi046.stdout:session_timeout 60 2024-04-27T04:02:19.944 INFO:teuthology.orchestra.run.smithi046.stdout:session_autoclose 300 2024-04-27T04:02:19.944 INFO:teuthology.orchestra.run.smithi046.stdout:max_file_size 1099511627776 2024-04-27T04:02:19.944 INFO:teuthology.orchestra.run.smithi046.stdout:max_xattr_size 65536 2024-04-27T04:02:19.944 INFO:teuthology.orchestra.run.smithi046.stdout:required_client_features {} 2024-04-27T04:02:19.944 INFO:teuthology.orchestra.run.smithi046.stdout:last_failure 0 2024-04-27T04:02:19.945 INFO:teuthology.orchestra.run.smithi046.stdout:last_failure_osd_epoch 46 2024-04-27T04:02:19.945 INFO:teuthology.orchestra.run.smithi046.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T04:02:19.945 INFO:teuthology.orchestra.run.smithi046.stdout:max_mds 2 2024-04-27T04:02:19.945 INFO:teuthology.orchestra.run.smithi046.stdout:in 0,1 2024-04-27T04:02:19.945 INFO:teuthology.orchestra.run.smithi046.stdout:up {0=14488,1=24285} 2024-04-27T04:02:19.946 INFO:teuthology.orchestra.run.smithi046.stdout:failed 2024-04-27T04:02:19.946 INFO:teuthology.orchestra.run.smithi046.stdout:damaged 2024-04-27T04:02:19.946 INFO:teuthology.orchestra.run.smithi046.stdout:stopped 2024-04-27T04:02:19.946 INFO:teuthology.orchestra.run.smithi046.stdout:data_pools [3] 2024-04-27T04:02:19.946 INFO:teuthology.orchestra.run.smithi046.stdout:metadata_pool 2 2024-04-27T04:02:19.947 INFO:teuthology.orchestra.run.smithi046.stdout:inline_data enabled 2024-04-27T04:02:19.947 INFO:teuthology.orchestra.run.smithi046.stdout:balancer 2024-04-27T04:02:19.947 INFO:teuthology.orchestra.run.smithi046.stdout:bal_rank_mask -1 2024-04-27T04:02:19.947 INFO:teuthology.orchestra.run.smithi046.stdout:standby_count_wanted 1 2024-04-27T04:02:19.947 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.cephfs.smithi046.mycakd{0:14488} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.46:6828/3081610305,v1:172.21.15.46:6829/3081610305] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T04:02:19.947 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.cephfs.smithi046.qgmaql{0:24303} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.46:6826/2653255940,v1:172.21.15.46:6827/2653255940] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T04:02:19.948 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.cephfs.smithi073.zkoypj{1:24285} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.73:6824/3397960510,v1:172.21.15.73:6825/3397960510] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T04:02:19.948 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.cephfs.smithi073.yryono{1:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.73:6826/565262827,v1:172.21.15.73:6827/565262827] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T04:02:19.948 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:02:19.948 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:02:19.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.941+0000 7fab1affd700 1 -- 172.21.15.46:0/3660237743 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fab1c06ccf0 msgr2=0x7fab1c06f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:19.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.942+0000 7fab1affd700 1 --2- 172.21.15.46:0/3660237743 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fab1c06ccf0 0x7fab1c06f1b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fab28005950 tx=0x7fab28005cf0 comp rx=0 tx=0).stop 2024-04-27T04:02:19.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.942+0000 7fab1affd700 1 -- 172.21.15.46:0/3660237743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab340ffb10 msgr2=0x7fab34069960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:19.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.942+0000 7fab1affd700 1 --2- 172.21.15.46:0/3660237743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab340ffb10 0x7fab34069960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fab2400bd70 tx=0x7fab2400be50 comp rx=0 tx=0).stop 2024-04-27T04:02:19.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.942+0000 7fab1affd700 1 -- 172.21.15.46:0/3660237743 shutdown_connections 2024-04-27T04:02:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.942+0000 7fab1affd700 1 --2- 172.21.15.46:0/3660237743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab340ffb10 0x7fab34069960 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.942+0000 7fab1affd700 1 --2- 172.21.15.46:0/3660237743 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fab1c06ccf0 0x7fab1c06f1b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.942+0000 7fab1affd700 1 --2- 172.21.15.46:0/3660237743 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab34100d10 0x7fab3419b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.942+0000 7fab1affd700 1 -- 172.21.15.46:0/3660237743 >> 172.21.15.46:0/3660237743 conn(0x7fab340fb0b0 msgr2=0x7fab341042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.942+0000 7fab1affd700 1 -- 172.21.15.46:0/3660237743 shutdown_connections 2024-04-27T04:02:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:19.942+0000 7fab1affd700 1 -- 172.21.15.46:0/3660237743 wait complete. 2024-04-27T04:02:19.952 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 19 2024-04-27T04:02:20.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.058+0000 7f462e9db700 1 -- 172.21.15.46:0/2026092890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4628104070 msgr2=0x7f4628106460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:20.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.058+0000 7f462e9db700 1 --2- 172.21.15.46:0/2026092890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4628104070 0x7f4628106460 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f461c009910 tx=0x7f461c009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:20.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.058+0000 7f462e9db700 1 -- 172.21.15.46:0/2026092890 shutdown_connections 2024-04-27T04:02:20.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.058+0000 7f462e9db700 1 --2- 172.21.15.46:0/2026092890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4628104070 0x7f4628106460 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:20.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.058+0000 7f462e9db700 1 --2- 172.21.15.46:0/2026092890 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4628101740 0x7f4628103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:20.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.058+0000 7f462e9db700 1 -- 172.21.15.46:0/2026092890 >> 172.21.15.46:0/2026092890 conn(0x7f46280fb0b0 msgr2=0x7f46280fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:20.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.058+0000 7f462e9db700 1 -- 172.21.15.46:0/2026092890 shutdown_connections 2024-04-27T04:02:20.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.058+0000 7f462e9db700 1 -- 172.21.15.46:0/2026092890 wait complete. 2024-04-27T04:02:20.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.059+0000 7f462e9db700 1 Processor -- start 2024-04-27T04:02:20.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.059+0000 7f462e9db700 1 -- start start 2024-04-27T04:02:20.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.059+0000 7f462e9db700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4628101740 0x7f4628101050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:20.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.059+0000 7f462e9db700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4628104070 0x7f46280ff6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:20.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.059+0000 7f462e9db700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46280ffc30 con 0x7f4628101740 2024-04-27T04:02:20.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.059+0000 7f462e9db700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46280ffd70 con 0x7f4628104070 2024-04-27T04:02:20.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.059+0000 7f4627fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4628101740 0x7f4628101050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:20.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.059+0000 7f4627fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4628101740 0x7f4628101050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38526/0 (socket says 172.21.15.46:38526) 2024-04-27T04:02:20.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.059+0000 7f4627fff700 1 -- 172.21.15.46:0/1409347911 learned_addr learned my addr 172.21.15.46:0/1409347911 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:20.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.060+0000 7f46277fe700 1 --2- 172.21.15.46:0/1409347911 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4628104070 0x7f46280ff6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:20.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.060+0000 7f4627fff700 1 -- 172.21.15.46:0/1409347911 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4628104070 msgr2=0x7f46280ff6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:20.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.060+0000 7f4627fff700 1 --2- 172.21.15.46:0/1409347911 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4628104070 0x7f46280ff6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:20.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.060+0000 7f4627fff700 1 -- 172.21.15.46:0/1409347911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461c0095f0 con 0x7f4628101740 2024-04-27T04:02:20.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.060+0000 7f4627fff700 1 --2- 172.21.15.46:0/1409347911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4628101740 0x7f4628101050 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f461800c8a0 tx=0x7f461800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:20.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.060+0000 7f46257fa700 1 -- 172.21.15.46:0/1409347911 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f461800cea0 con 0x7f4628101740 2024-04-27T04:02:20.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.060+0000 7f462e9db700 1 -- 172.21.15.46:0/1409347911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4628100050 con 0x7f4628101740 2024-04-27T04:02:20.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.060+0000 7f46257fa700 1 -- 172.21.15.46:0/1409347911 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4618004830 con 0x7f4628101740 2024-04-27T04:02:20.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.060+0000 7f462e9db700 1 -- 172.21.15.46:0/1409347911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462819e650 con 0x7f4628101740 2024-04-27T04:02:20.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.061+0000 7f46257fa700 1 -- 172.21.15.46:0/1409347911 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46180055d0 con 0x7f4628101740 2024-04-27T04:02:20.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.061+0000 7f462e9db700 1 -- 172.21.15.46:0/1409347911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462804ebb0 con 0x7f4628101740 2024-04-27T04:02:20.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.062+0000 7f46257fa700 1 -- 172.21.15.46:0/1409347911 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f46180049a0 con 0x7f4628101740 2024-04-27T04:02:20.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.062+0000 7f46257fa700 1 --2- 172.21.15.46:0/1409347911 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f461006cce0 0x7f461006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:20.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.063+0000 7f46257fa700 1 -- 172.21.15.46:0/1409347911 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f461808c2f0 con 0x7f4628101740 2024-04-27T04:02:20.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.063+0000 7f46277fe700 1 --2- 172.21.15.46:0/1409347911 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f461006cce0 0x7f461006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:20.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.063+0000 7f46277fe700 1 --2- 172.21.15.46:0/1409347911 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f461006cce0 0x7f461006f1a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f461c0051c0 tx=0x7f461c0058e0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:20.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.067+0000 7f46257fa700 1 -- 172.21.15.46:0/1409347911 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f461805a0f0 con 0x7f4628101740 2024-04-27T04:02:20.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.305+0000 7f462e9db700 1 -- 172.21.15.46:0/1409347911 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4628105f90 con 0x7f461006cce0 2024-04-27T04:02:20.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.307+0000 7f46257fa700 1 -- 172.21.15.46:0/1409347911 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f4628105f90 con 0x7f461006cce0 2024-04-27T04:02:20.307 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-27T04:02:20.307 INFO:teuthology.orchestra.run.smithi046.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:4e213a58e787437ff508e6c890ccd5fd32dc96b5", 2024-04-27T04:02:20.307 INFO:teuthology.orchestra.run.smithi046.stdout: "in_progress": true, 2024-04-27T04:02:20.308 INFO:teuthology.orchestra.run.smithi046.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T04:02:20.308 INFO:teuthology.orchestra.run.smithi046.stdout: "services_complete": [], 2024-04-27T04:02:20.308 INFO:teuthology.orchestra.run.smithi046.stdout: "progress": "", 2024-04-27T04:02:20.308 INFO:teuthology.orchestra.run.smithi046.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:4e213a58e787437ff508e6c890ccd5fd32dc96b5 image", 2024-04-27T04:02:20.308 INFO:teuthology.orchestra.run.smithi046.stdout: "is_paused": false 2024-04-27T04:02:20.308 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-27T04:02:20.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.309+0000 7f462e9db700 1 -- 172.21.15.46:0/1409347911 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f461006cce0 msgr2=0x7f461006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:20.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.309+0000 7f462e9db700 1 --2- 172.21.15.46:0/1409347911 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f461006cce0 0x7f461006f1a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f461c0051c0 tx=0x7f461c0058e0 comp rx=0 tx=0).stop 2024-04-27T04:02:20.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.309+0000 7f462e9db700 1 -- 172.21.15.46:0/1409347911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4628101740 msgr2=0x7f4628101050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.309+0000 7f462e9db700 1 --2- 172.21.15.46:0/1409347911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4628101740 0x7f4628101050 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f461800c8a0 tx=0x7f461800cc60 comp rx=0 tx=0).stop 2024-04-27T04:02:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.310+0000 7f462e9db700 1 -- 172.21.15.46:0/1409347911 shutdown_connections 2024-04-27T04:02:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.310+0000 7f462e9db700 1 --2- 172.21.15.46:0/1409347911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4628101740 0x7f4628101050 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.310+0000 7f462e9db700 1 --2- 172.21.15.46:0/1409347911 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f461006cce0 0x7f461006f1a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.310+0000 7f462e9db700 1 --2- 172.21.15.46:0/1409347911 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4628104070 0x7f46280ff6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.310+0000 7f462e9db700 1 -- 172.21.15.46:0/1409347911 >> 172.21.15.46:0/1409347911 conn(0x7f46280fb0b0 msgr2=0x7f46280fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.310+0000 7f462e9db700 1 -- 172.21.15.46:0/1409347911 shutdown_connections 2024-04-27T04:02:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.310+0000 7f462e9db700 1 -- 172.21.15.46:0/1409347911 wait complete. 2024-04-27T04:02:20.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.425+0000 7fe6f0307700 1 -- 172.21.15.46:0/4140486228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6e8100d90 msgr2=0x7fe6e81011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:20.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.425+0000 7fe6f0307700 1 --2- 172.21.15.46:0/4140486228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6e8100d90 0x7fe6e81011f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8009910 tx=0x7fe6d8009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:20.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.425+0000 7fe6f0307700 1 -- 172.21.15.46:0/4140486228 shutdown_connections 2024-04-27T04:02:20.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.425+0000 7fe6f0307700 1 --2- 172.21.15.46:0/4140486228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6e8100d90 0x7fe6e81011f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:20.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.425+0000 7fe6f0307700 1 --2- 172.21.15.46:0/4140486228 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe6e80ffb90 0x7fe6e80fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:20.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.425+0000 7fe6f0307700 1 -- 172.21.15.46:0/4140486228 >> 172.21.15.46:0/4140486228 conn(0x7fe6e80fb110 msgr2=0x7fe6e80fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:20.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.425+0000 7fe6f0307700 1 -- 172.21.15.46:0/4140486228 shutdown_connections 2024-04-27T04:02:20.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.425+0000 7fe6f0307700 1 -- 172.21.15.46:0/4140486228 wait complete. 2024-04-27T04:02:20.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.426+0000 7fe6f0307700 1 Processor -- start 2024-04-27T04:02:20.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.426+0000 7fe6f0307700 1 -- start start 2024-04-27T04:02:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.426+0000 7fe6f0307700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6e80ffb90 0x7fe6e8196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.426+0000 7fe6f0307700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe6e8100d90 0x7fe6e8196cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.426+0000 7fe6f0307700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e81972f0 con 0x7fe6e80ffb90 2024-04-27T04:02:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.426+0000 7fe6f0307700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e8197430 con 0x7fe6e8100d90 2024-04-27T04:02:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.426+0000 7fe6ee0a3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6e80ffb90 0x7fe6e8196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.426+0000 7fe6ee0a3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6e80ffb90 0x7fe6e8196790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38530/0 (socket says 172.21.15.46:38530) 2024-04-27T04:02:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.426+0000 7fe6ee0a3700 1 -- 172.21.15.46:0/2234197457 learned_addr learned my addr 172.21.15.46:0/2234197457 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.426+0000 7fe6ed8a2700 1 --2- 172.21.15.46:0/2234197457 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe6e8100d90 0x7fe6e8196cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.426+0000 7fe6ee0a3700 1 -- 172.21.15.46:0/2234197457 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe6e8100d90 msgr2=0x7fe6e8196cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.426+0000 7fe6ee0a3700 1 --2- 172.21.15.46:0/2234197457 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe6e8100d90 0x7fe6e8196cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.426+0000 7fe6ee0a3700 1 -- 172.21.15.46:0/2234197457 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d80095f0 con 0x7fe6e80ffb90 2024-04-27T04:02:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.427+0000 7fe6ee0a3700 1 --2- 172.21.15.46:0/2234197457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6e80ffb90 0x7fe6e8196790 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe6e400b480 tx=0x7fe6e400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.427+0000 7fe6df7fe700 1 -- 172.21.15.46:0/2234197457 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6e40118c0 con 0x7fe6e80ffb90 2024-04-27T04:02:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.427+0000 7fe6df7fe700 1 -- 172.21.15.46:0/2234197457 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6e4010380 con 0x7fe6e80ffb90 2024-04-27T04:02:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.427+0000 7fe6f0307700 1 -- 172.21.15.46:0/2234197457 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e819be40 con 0x7fe6e80ffb90 2024-04-27T04:02:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.427+0000 7fe6df7fe700 1 -- 172.21.15.46:0/2234197457 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6e400f4f0 con 0x7fe6e80ffb90 2024-04-27T04:02:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.427+0000 7fe6f0307700 1 -- 172.21.15.46:0/2234197457 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e819c380 con 0x7fe6e80ffb90 2024-04-27T04:02:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.428+0000 7fe6f0307700 1 -- 172.21.15.46:0/2234197457 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e804ebb0 con 0x7fe6e80ffb90 2024-04-27T04:02:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.429+0000 7fe6df7fe700 1 -- 172.21.15.46:0/2234197457 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe6e40104f0 con 0x7fe6e80ffb90 2024-04-27T04:02:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.429+0000 7fe6df7fe700 1 --2- 172.21.15.46:0/2234197457 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe6d406cce0 0x7fe6d406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.429+0000 7fe6df7fe700 1 -- 172.21.15.46:0/2234197457 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fe6e408be10 con 0x7fe6e80ffb90 2024-04-27T04:02:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.429+0000 7fe6ed8a2700 1 --2- 172.21.15.46:0/2234197457 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe6d406cce0 0x7fe6d406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.430+0000 7fe6ed8a2700 1 --2- 172.21.15.46:0/2234197457 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe6d406cce0 0x7fe6d406f1a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8006010 tx=0x7fe6d800b580 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.434+0000 7fe6df7fe700 1 -- 172.21.15.46:0/2234197457 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fe6e4059c90 con 0x7fe6e80ffb90 2024-04-27T04:02:20.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:20 smithi046 ceph-mon[26899]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:20.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:20 smithi046 ceph-mon[26899]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-27T04:02:20.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:20 smithi046 ceph-mon[26899]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:20.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:20 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2990771200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T04:02:20.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:20 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/3660237743' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T04:02:20.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.738+0000 7fe6f0307700 1 -- 172.21.15.46:0/2234197457 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe6e8108790 con 0x7fe6e80ffb90 2024-04-27T04:02:20.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.738+0000 7fe6df7fe700 1 -- 172.21.15.46:0/2234197457 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fe6e4059820 con 0x7fe6e80ffb90 2024-04-27T04:02:20.739 INFO:teuthology.orchestra.run.smithi046.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-27T04:02:20.739 INFO:teuthology.orchestra.run.smithi046.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-27T04:02:20.739 INFO:teuthology.orchestra.run.smithi046.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-27T04:02:20.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.740+0000 7fe6f0307700 1 -- 172.21.15.46:0/2234197457 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe6d406cce0 msgr2=0x7fe6d406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:20.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.740+0000 7fe6f0307700 1 --2- 172.21.15.46:0/2234197457 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe6d406cce0 0x7fe6d406f1a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8006010 tx=0x7fe6d800b580 comp rx=0 tx=0).stop 2024-04-27T04:02:20.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.740+0000 7fe6f0307700 1 -- 172.21.15.46:0/2234197457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6e80ffb90 msgr2=0x7fe6e8196790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:20.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.740+0000 7fe6f0307700 1 --2- 172.21.15.46:0/2234197457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6e80ffb90 0x7fe6e8196790 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe6e400b480 tx=0x7fe6e400b840 comp rx=0 tx=0).stop 2024-04-27T04:02:20.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.741+0000 7fe6f0307700 1 -- 172.21.15.46:0/2234197457 shutdown_connections 2024-04-27T04:02:20.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.741+0000 7fe6f0307700 1 --2- 172.21.15.46:0/2234197457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6e80ffb90 0x7fe6e8196790 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:20.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.741+0000 7fe6f0307700 1 --2- 172.21.15.46:0/2234197457 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fe6d406cce0 0x7fe6d406f1a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:20.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.741+0000 7fe6f0307700 1 --2- 172.21.15.46:0/2234197457 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe6e8100d90 0x7fe6e8196cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:20.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.741+0000 7fe6f0307700 1 -- 172.21.15.46:0/2234197457 >> 172.21.15.46:0/2234197457 conn(0x7fe6e80fb110 msgr2=0x7fe6e8068670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:20.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.741+0000 7fe6f0307700 1 -- 172.21.15.46:0/2234197457 shutdown_connections 2024-04-27T04:02:20.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:20.741+0000 7fe6f0307700 1 -- 172.21.15.46:0/2234197457 wait complete. 2024-04-27T04:02:20.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:20 smithi073 ceph-mon[34160]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:20.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:20 smithi073 ceph-mon[34160]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-27T04:02:20.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:20 smithi073 ceph-mon[34160]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:20.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:20 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/2990771200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T04:02:20.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:20 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/3660237743' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T04:02:21.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:21 smithi046 ceph-mon[26899]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:21.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:21 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2234197457' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T04:02:21.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:21 smithi073 ceph-mon[34160]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:21.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:21 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/2234197457' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T04:02:22.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:22 smithi046 ceph-mon[26899]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-27T04:02:22.761 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:22 smithi073 ceph-mon[34160]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-27T04:02:24.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:24 smithi046 ceph-mon[26899]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:02:24.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:24 smithi073 ceph-mon[34160]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:02:26.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:26 smithi046 ceph-mon[26899]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:02:26.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:26 smithi073 ceph-mon[34160]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:02:28.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:28 smithi046 ceph-mon[26899]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:02:28.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:28 smithi073 ceph-mon[34160]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:02:30.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:30 smithi046 ceph-mon[26899]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:02:30.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:30 smithi073 ceph-mon[34160]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:02:31.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:02:31.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:31 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:02:32.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:32 smithi046 ceph-mon[26899]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-27T04:02:32.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:32 smithi073 ceph-mon[34160]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-27T04:02:34.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:34 smithi046 ceph-mon[26899]: pgmap v215: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:02:34.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:34 smithi073 ceph-mon[34160]: pgmap v215: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:02:36.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:36 smithi046 ceph-mon[26899]: pgmap v216: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:02:36.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:36 smithi073 ceph-mon[34160]: pgmap v216: 65 pgs: 65 active+clean; 586 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:02:38.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:38 smithi046 ceph-mon[26899]: pgmap v217: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-27T04:02:38.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:38 smithi073 ceph-mon[34160]: pgmap v217: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-27T04:02:40.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:40 smithi046 ceph-mon[26899]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-27T04:02:40.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:40 smithi073 ceph-mon[34160]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-27T04:02:42.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:42 smithi046 ceph-mon[26899]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 426 B/s wr, 3 op/s 2024-04-27T04:02:42.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:42 smithi073 ceph-mon[34160]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 426 B/s wr, 3 op/s 2024-04-27T04:02:44.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:44 smithi046 ceph-mon[26899]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-27T04:02:44.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:44 smithi073 ceph-mon[34160]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-27T04:02:46.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:46 smithi046 ceph-mon[26899]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-27T04:02:46.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:46 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:02:46.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:46 smithi073 ceph-mon[34160]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-27T04:02:46.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:46 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:02:48.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:48 smithi046 ceph-mon[26899]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-27T04:02:48.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:48 smithi073 ceph-mon[34160]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-27T04:02:50.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:49 smithi073 ceph-mon[34160]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-27T04:02:50.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:49 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:02:50.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:49 smithi073 ceph-mon[34160]: Upgrade: Target is version 19.0.0-3253-g4e213a58 (unknown) 2024-04-27T04:02:50.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:49 smithi073 ceph-mon[34160]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:90a3a9fa9ceb8977f6dad9ccd36fe46018d07e2118199820523a6b35301d9c16, digests ['quay.ceph.io/ceph-ci/ceph@sha256:90a3a9fa9ceb8977f6dad9ccd36fe46018d07e2118199820523a6b35301d9c16'] 2024-04-27T04:02:50.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:49 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:02:50.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:49 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:02:50.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:49 smithi073 ceph-mon[34160]: Upgrade: Need to upgrade myself (mgr.smithi046.qrfdvr) 2024-04-27T04:02:50.076 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:49 smithi073 ceph-mon[34160]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:90a3a9fa9ceb8977f6dad9ccd36fe46018d07e2118199820523a6b35301d9c16 on smithi073 2024-04-27T04:02:50.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:49 smithi046 ceph-mon[26899]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-27T04:02:50.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:49 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:02:50.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:49 smithi046 ceph-mon[26899]: Upgrade: Target is version 19.0.0-3253-g4e213a58 (unknown) 2024-04-27T04:02:50.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:49 smithi046 ceph-mon[26899]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:90a3a9fa9ceb8977f6dad9ccd36fe46018d07e2118199820523a6b35301d9c16, digests ['quay.ceph.io/ceph-ci/ceph@sha256:90a3a9fa9ceb8977f6dad9ccd36fe46018d07e2118199820523a6b35301d9c16'] 2024-04-27T04:02:50.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:49 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:02:50.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:49 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:02:50.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:49 smithi046 ceph-mon[26899]: Upgrade: Need to upgrade myself (mgr.smithi046.qrfdvr) 2024-04-27T04:02:50.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:49 smithi046 ceph-mon[26899]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:90a3a9fa9ceb8977f6dad9ccd36fe46018d07e2118199820523a6b35301d9c16 on smithi073 2024-04-27T04:02:50.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.887+0000 7fcd775a3700 1 -- 172.21.15.46:0/1978575609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd70101720 msgr2=0x7fcd70103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:50.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.887+0000 7fcd775a3700 1 --2- 172.21.15.46:0/1978575609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd70101720 0x7fcd70103b10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c009910 tx=0x7fcd6c009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:50.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.887+0000 7fcd775a3700 1 -- 172.21.15.46:0/1978575609 shutdown_connections 2024-04-27T04:02:50.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.887+0000 7fcd775a3700 1 --2- 172.21.15.46:0/1978575609 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd70104050 0x7fcd70106440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:50.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.887+0000 7fcd775a3700 1 --2- 172.21.15.46:0/1978575609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd70101720 0x7fcd70103b10 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:50.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.887+0000 7fcd775a3700 1 -- 172.21.15.46:0/1978575609 >> 172.21.15.46:0/1978575609 conn(0x7fcd700fb110 msgr2=0x7fcd700fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:50.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.887+0000 7fcd775a3700 1 -- 172.21.15.46:0/1978575609 shutdown_connections 2024-04-27T04:02:50.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.887+0000 7fcd775a3700 1 -- 172.21.15.46:0/1978575609 wait complete. 2024-04-27T04:02:50.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.888+0000 7fcd775a3700 1 Processor -- start 2024-04-27T04:02:50.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.888+0000 7fcd775a3700 1 -- start start 2024-04-27T04:02:50.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.888+0000 7fcd775a3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd70101720 0x7fcd70196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:50.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.888+0000 7fcd775a3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd70104050 0x7fcd70196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:50.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.888+0000 7fcd775a3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd70197180 con 0x7fcd70104050 2024-04-27T04:02:50.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.888+0000 7fcd775a3700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd701972c0 con 0x7fcd70101720 2024-04-27T04:02:50.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.888+0000 7fcd74b3e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd70104050 0x7fcd70196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:50.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.888+0000 7fcd7533f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd70101720 0x7fcd70196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:50.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.888+0000 7fcd7533f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd70101720 0x7fcd70196620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.46:53476/0 (socket says 172.21.15.46:53476) 2024-04-27T04:02:50.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.888+0000 7fcd7533f700 1 -- 172.21.15.46:0/507754970 learned_addr learned my addr 172.21.15.46:0/507754970 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:50.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.889+0000 7fcd74b3e700 1 -- 172.21.15.46:0/507754970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd70101720 msgr2=0x7fcd70196620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:50.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.889+0000 7fcd74b3e700 1 --2- 172.21.15.46:0/507754970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd70101720 0x7fcd70196620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:50.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.889+0000 7fcd74b3e700 1 -- 172.21.15.46:0/507754970 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c0095f0 con 0x7fcd70104050 2024-04-27T04:02:50.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.889+0000 7fcd7533f700 1 --2- 172.21.15.46:0/507754970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd70101720 0x7fcd70196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T04:02:50.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.889+0000 7fcd74b3e700 1 --2- 172.21.15.46:0/507754970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd70104050 0x7fcd70196b60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000e970 tx=0x7fcd6000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:50.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.889+0000 7fcd667fc700 1 -- 172.21.15.46:0/507754970 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd6000c9d0 con 0x7fcd70104050 2024-04-27T04:02:50.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.890+0000 7fcd667fc700 1 -- 172.21.15.46:0/507754970 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd60010430 con 0x7fcd70104050 2024-04-27T04:02:50.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.890+0000 7fcd667fc700 1 -- 172.21.15.46:0/507754970 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd60018630 con 0x7fcd70104050 2024-04-27T04:02:50.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.890+0000 7fcd775a3700 1 -- 172.21.15.46:0/507754970 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7019bd20 con 0x7fcd70104050 2024-04-27T04:02:50.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.890+0000 7fcd775a3700 1 -- 172.21.15.46:0/507754970 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7019c220 con 0x7fcd70104050 2024-04-27T04:02:50.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.891+0000 7fcd775a3700 1 -- 172.21.15.46:0/507754970 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7004ebb0 con 0x7fcd70104050 2024-04-27T04:02:50.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.891+0000 7fcd667fc700 1 -- 172.21.15.46:0/507754970 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fcd600105a0 con 0x7fcd70104050 2024-04-27T04:02:50.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.892+0000 7fcd667fc700 1 --2- 172.21.15.46:0/507754970 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fcd5c06ccf0 0x7fcd5c06f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:50.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.892+0000 7fcd667fc700 1 -- 172.21.15.46:0/507754970 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fcd60014070 con 0x7fcd70104050 2024-04-27T04:02:50.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.892+0000 7fcd7533f700 1 --2- 172.21.15.46:0/507754970 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fcd5c06ccf0 0x7fcd5c06f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:50.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.893+0000 7fcd7533f700 1 --2- 172.21.15.46:0/507754970 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fcd5c06ccf0 0x7fcd5c06f1b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c00b6f0 tx=0x7fcd6c005a90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:50.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:50.901+0000 7fcd667fc700 1 -- 172.21.15.46:0/507754970 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fcd60052c50 con 0x7fcd70104050 2024-04-27T04:02:51.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.226+0000 7fcd775a3700 1 -- 172.21.15.46:0/507754970 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd70105f00 con 0x7fcd5c06ccf0 2024-04-27T04:02:51.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.227+0000 7fcd667fc700 1 -- 172.21.15.46:0/507754970 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7fcd70105f00 con 0x7fcd5c06ccf0 2024-04-27T04:02:51.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.229+0000 7fcd775a3700 1 -- 172.21.15.46:0/507754970 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fcd5c06ccf0 msgr2=0x7fcd5c06f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:51.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.229+0000 7fcd775a3700 1 --2- 172.21.15.46:0/507754970 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fcd5c06ccf0 0x7fcd5c06f1b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c00b6f0 tx=0x7fcd6c005a90 comp rx=0 tx=0).stop 2024-04-27T04:02:51.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.229+0000 7fcd775a3700 1 -- 172.21.15.46:0/507754970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd70104050 msgr2=0x7fcd70196b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:51.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.229+0000 7fcd775a3700 1 --2- 172.21.15.46:0/507754970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd70104050 0x7fcd70196b60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000e970 tx=0x7fcd6000ec80 comp rx=0 tx=0).stop 2024-04-27T04:02:51.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.229+0000 7fcd775a3700 1 -- 172.21.15.46:0/507754970 shutdown_connections 2024-04-27T04:02:51.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.229+0000 7fcd775a3700 1 --2- 172.21.15.46:0/507754970 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fcd5c06ccf0 0x7fcd5c06f1b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:51.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.229+0000 7fcd775a3700 1 --2- 172.21.15.46:0/507754970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd70101720 0x7fcd70196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:51.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.229+0000 7fcd775a3700 1 --2- 172.21.15.46:0/507754970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd70104050 0x7fcd70196b60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:51.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.229+0000 7fcd775a3700 1 -- 172.21.15.46:0/507754970 >> 172.21.15.46:0/507754970 conn(0x7fcd700fb110 msgr2=0x7fcd700fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:51.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.229+0000 7fcd775a3700 1 -- 172.21.15.46:0/507754970 shutdown_connections 2024-04-27T04:02:51.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.229+0000 7fcd775a3700 1 -- 172.21.15.46:0/507754970 wait complete. 2024-04-27T04:02:51.240 INFO:teuthology.orchestra.run.smithi046.stdout:true 2024-04-27T04:02:51.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.372+0000 7f4f30dd2700 1 -- 172.21.15.46:0/1384243538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f2c100c40 msgr2=0x7f4f2c101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:51.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.372+0000 7f4f30dd2700 1 --2- 172.21.15.46:0/1384243538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f2c100c40 0x7f4f2c101060 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c009910 tx=0x7f4f1c009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:51.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.372+0000 7f4f30dd2700 1 -- 172.21.15.46:0/1384243538 shutdown_connections 2024-04-27T04:02:51.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.372+0000 7f4f30dd2700 1 --2- 172.21.15.46:0/1384243538 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f2c101e40 0x7f4f2c1022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:51.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.372+0000 7f4f30dd2700 1 --2- 172.21.15.46:0/1384243538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f2c100c40 0x7f4f2c101060 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:51.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.372+0000 7f4f30dd2700 1 -- 172.21.15.46:0/1384243538 >> 172.21.15.46:0/1384243538 conn(0x7f4f2c0fc200 msgr2=0x7f4f2c0fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:51.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.372+0000 7f4f30dd2700 1 -- 172.21.15.46:0/1384243538 shutdown_connections 2024-04-27T04:02:51.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.372+0000 7f4f30dd2700 1 -- 172.21.15.46:0/1384243538 wait complete. 2024-04-27T04:02:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.373+0000 7f4f30dd2700 1 Processor -- start 2024-04-27T04:02:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.373+0000 7f4f30dd2700 1 -- start start 2024-04-27T04:02:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.373+0000 7f4f30dd2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f2c100c40 0x7f4f2c074be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.373+0000 7f4f30dd2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f2c101e40 0x7f4f2c0732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.373+0000 7f4f30dd2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c073810 con 0x7f4f2c101e40 2024-04-27T04:02:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.373+0000 7f4f30dd2700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c073950 con 0x7f4f2c100c40 2024-04-27T04:02:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.373+0000 7f4f29d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f2c101e40 0x7f4f2c0732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.373+0000 7f4f2a59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f2c100c40 0x7f4f2c074be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.373+0000 7f4f29d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f2c101e40 0x7f4f2c0732d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58340/0 (socket says 172.21.15.46:58340) 2024-04-27T04:02:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.373+0000 7f4f2a59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f2c100c40 0x7f4f2c074be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.46:53492/0 (socket says 172.21.15.46:53492) 2024-04-27T04:02:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.373+0000 7f4f29d9b700 1 -- 172.21.15.46:0/2036691124 learned_addr learned my addr 172.21.15.46:0/2036691124 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.374+0000 7f4f29d9b700 1 -- 172.21.15.46:0/2036691124 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f2c100c40 msgr2=0x7f4f2c074be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.374+0000 7f4f29d9b700 1 --2- 172.21.15.46:0/2036691124 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f2c100c40 0x7f4f2c074be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.374+0000 7f4f29d9b700 1 -- 172.21.15.46:0/2036691124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c0095f0 con 0x7f4f2c101e40 2024-04-27T04:02:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.374+0000 7f4f29d9b700 1 --2- 172.21.15.46:0/2036691124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f2c101e40 0x7f4f2c0732d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000e920 tx=0x7f4f2000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.374+0000 7f4f1b7fe700 1 -- 172.21.15.46:0/2036691124 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f2000c9d0 con 0x7f4f2c101e40 2024-04-27T04:02:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.374+0000 7f4f1b7fe700 1 -- 172.21.15.46:0/2036691124 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f20010430 con 0x7f4f2c101e40 2024-04-27T04:02:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.374+0000 7f4f30dd2700 1 -- 172.21.15.46:0/2036691124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2c073c30 con 0x7f4f2c101e40 2024-04-27T04:02:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.374+0000 7f4f1b7fe700 1 -- 172.21.15.46:0/2036691124 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f20018630 con 0x7f4f2c101e40 2024-04-27T04:02:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.375+0000 7f4f30dd2700 1 -- 172.21.15.46:0/2036691124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2c19e6c0 con 0x7f4f2c101e40 2024-04-27T04:02:51.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.376+0000 7f4f30dd2700 1 -- 172.21.15.46:0/2036691124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2c1907c0 con 0x7f4f2c101e40 2024-04-27T04:02:51.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.377+0000 7f4f1b7fe700 1 -- 172.21.15.46:0/2036691124 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4f200105a0 con 0x7f4f2c101e40 2024-04-27T04:02:51.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.377+0000 7f4f1b7fe700 1 --2- 172.21.15.46:0/2036691124 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f4f1406ccf0 0x7f4f1406f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:51.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.377+0000 7f4f1b7fe700 1 -- 172.21.15.46:0/2036691124 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f4f20014070 con 0x7f4f2c101e40 2024-04-27T04:02:51.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.378+0000 7f4f2a59c700 1 --2- 172.21.15.46:0/2036691124 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f4f1406ccf0 0x7f4f1406f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:51.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.378+0000 7f4f2a59c700 1 --2- 172.21.15.46:0/2036691124 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f4f1406ccf0 0x7f4f1406f1b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00b800 tx=0x7f4f1c009f70 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:51.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.382+0000 7f4f1b7fe700 1 -- 172.21.15.46:0/2036691124 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f4f20052c50 con 0x7f4f2c101e40 2024-04-27T04:02:51.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.760+0000 7f4f30dd2700 1 -- 172.21.15.46:0/2036691124 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f2c02cdb0 con 0x7f4f1406ccf0 2024-04-27T04:02:51.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.762+0000 7f4f1b7fe700 1 -- 172.21.15.46:0/2036691124 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f4f2c02cdb0 con 0x7f4f1406ccf0 2024-04-27T04:02:51.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.764+0000 7f4f30dd2700 1 -- 172.21.15.46:0/2036691124 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f4f1406ccf0 msgr2=0x7f4f1406f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:51.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.764+0000 7f4f30dd2700 1 --2- 172.21.15.46:0/2036691124 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f4f1406ccf0 0x7f4f1406f1b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00b800 tx=0x7f4f1c009f70 comp rx=0 tx=0).stop 2024-04-27T04:02:51.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.764+0000 7f4f30dd2700 1 -- 172.21.15.46:0/2036691124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f2c101e40 msgr2=0x7f4f2c0732d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:51.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.764+0000 7f4f30dd2700 1 --2- 172.21.15.46:0/2036691124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f2c101e40 0x7f4f2c0732d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000e920 tx=0x7f4f2000ec30 comp rx=0 tx=0).stop 2024-04-27T04:02:51.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.764+0000 7f4f30dd2700 1 -- 172.21.15.46:0/2036691124 shutdown_connections 2024-04-27T04:02:51.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.764+0000 7f4f30dd2700 1 --2- 172.21.15.46:0/2036691124 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f4f1406ccf0 0x7f4f1406f1b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:51.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.764+0000 7f4f30dd2700 1 --2- 172.21.15.46:0/2036691124 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f2c100c40 0x7f4f2c074be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:51.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.764+0000 7f4f30dd2700 1 --2- 172.21.15.46:0/2036691124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f2c101e40 0x7f4f2c0732d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:51.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.764+0000 7f4f30dd2700 1 -- 172.21.15.46:0/2036691124 >> 172.21.15.46:0/2036691124 conn(0x7f4f2c0fc200 msgr2=0x7f4f2c105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:51.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.764+0000 7f4f30dd2700 1 -- 172.21.15.46:0/2036691124 shutdown_connections 2024-04-27T04:02:51.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.764+0000 7f4f30dd2700 1 -- 172.21.15.46:0/2036691124 wait complete. 2024-04-27T04:02:51.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.885+0000 7f84e3d17700 1 -- 172.21.15.46:0/3450656290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84dc1040d0 msgr2=0x7f84dc1064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:51.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.885+0000 7f84e3d17700 1 --2- 172.21.15.46:0/3450656290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84dc1040d0 0x7f84dc1064c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f84cc009910 tx=0x7f84cc009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:51.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.885+0000 7f84e3d17700 1 -- 172.21.15.46:0/3450656290 shutdown_connections 2024-04-27T04:02:51.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.885+0000 7f84e3d17700 1 --2- 172.21.15.46:0/3450656290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84dc1040d0 0x7f84dc1064c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:51.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.885+0000 7f84e3d17700 1 --2- 172.21.15.46:0/3450656290 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84dc1017a0 0x7f84dc103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:51.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.885+0000 7f84e3d17700 1 -- 172.21.15.46:0/3450656290 >> 172.21.15.46:0/3450656290 conn(0x7f84dc0fb110 msgr2=0x7f84dc0fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:51.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.885+0000 7f84e3d17700 1 -- 172.21.15.46:0/3450656290 shutdown_connections 2024-04-27T04:02:51.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.885+0000 7f84e3d17700 1 -- 172.21.15.46:0/3450656290 wait complete. 2024-04-27T04:02:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.886+0000 7f84e3d17700 1 Processor -- start 2024-04-27T04:02:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.886+0000 7f84e3d17700 1 -- start start 2024-04-27T04:02:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.886+0000 7f84e3d17700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84dc1017a0 0x7f84dc196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.886+0000 7f84e3d17700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84dc1040d0 0x7f84dc196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.886+0000 7f84e3d17700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc197160 con 0x7f84dc1017a0 2024-04-27T04:02:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.886+0000 7f84e3d17700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc1972a0 con 0x7f84dc1040d0 2024-04-27T04:02:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.886+0000 7f84e1ab3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84dc1017a0 0x7f84dc196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.886+0000 7f84e1ab3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84dc1017a0 0x7f84dc196600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58364/0 (socket says 172.21.15.46:58364) 2024-04-27T04:02:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.886+0000 7f84e1ab3700 1 -- 172.21.15.46:0/1647864543 learned_addr learned my addr 172.21.15.46:0/1647864543 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.886+0000 7f84e12b2700 1 --2- 172.21.15.46:0/1647864543 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84dc1040d0 0x7f84dc196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.887+0000 7f84e1ab3700 1 -- 172.21.15.46:0/1647864543 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84dc1040d0 msgr2=0x7f84dc196b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.887+0000 7f84e1ab3700 1 --2- 172.21.15.46:0/1647864543 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84dc1040d0 0x7f84dc196b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.887+0000 7f84e1ab3700 1 -- 172.21.15.46:0/1647864543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84cc0095f0 con 0x7f84dc1017a0 2024-04-27T04:02:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.887+0000 7f84e1ab3700 1 --2- 172.21.15.46:0/1647864543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84dc1017a0 0x7f84dc196600 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009cb0 tx=0x7f84d800b2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:51.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.887+0000 7f84d2ffd700 1 -- 172.21.15.46:0/1647864543 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84d8015070 con 0x7f84dc1017a0 2024-04-27T04:02:51.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.887+0000 7f84d2ffd700 1 -- 172.21.15.46:0/1647864543 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84d800bd50 con 0x7f84dc1017a0 2024-04-27T04:02:51.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.888+0000 7f84d2ffd700 1 -- 172.21.15.46:0/1647864543 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84d8018770 con 0x7f84dc1017a0 2024-04-27T04:02:51.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.888+0000 7f84e3d17700 1 -- 172.21.15.46:0/1647864543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84dc0ff5f0 con 0x7f84dc1017a0 2024-04-27T04:02:51.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.888+0000 7f84e3d17700 1 -- 172.21.15.46:0/1647864543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84dc0ffac0 con 0x7f84dc1017a0 2024-04-27T04:02:51.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.890+0000 7f84e3d17700 1 -- 172.21.15.46:0/1647864543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84dc04ebb0 con 0x7f84dc1017a0 2024-04-27T04:02:51.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.890+0000 7f84d2ffd700 1 -- 172.21.15.46:0/1647864543 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f84d80188d0 con 0x7f84dc1017a0 2024-04-27T04:02:51.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.891+0000 7f84d2ffd700 1 --2- 172.21.15.46:0/1647864543 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f84c806cb30 0x7f84c806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:51.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.891+0000 7f84d2ffd700 1 -- 172.21.15.46:0/1647864543 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f84d808fbb0 con 0x7f84dc1017a0 2024-04-27T04:02:51.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.891+0000 7f84e12b2700 1 --2- 172.21.15.46:0/1647864543 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f84c806cb30 0x7f84c806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:51.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.892+0000 7f84e12b2700 1 --2- 172.21.15.46:0/1647864543 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f84c806cb30 0x7f84c806eff0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00b760 tx=0x7f84cc009f70 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:51.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:51.899+0000 7f84d2ffd700 1 -- 172.21.15.46:0/1647864543 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f84d805d910 con 0x7f84dc1017a0 2024-04-27T04:02:52.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:51 smithi046 ceph-mon[26899]: pgmap v224: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-04-27T04:02:52.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:51 smithi046 ceph-mon[26899]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:52.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.203+0000 7f84e3d17700 1 -- 172.21.15.46:0/1647864543 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f84dc105f60 con 0x7f84c806cb30 2024-04-27T04:02:52.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.224+0000 7f84d2ffd700 1 -- 172.21.15.46:0/1647864543 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f84dc105f60 con 0x7f84c806cb30 2024-04-27T04:02:52.224 INFO:teuthology.orchestra.run.smithi046.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T04:02:52.224 INFO:teuthology.orchestra.run.smithi046.stdout:alertmanager.smithi046 smithi046 *:9093,9094 running (5m) 60s ago 6m 24.8M - 0.25.0 c8568f914cd2 698cffee765e 2024-04-27T04:02:52.224 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-exporter.smithi046 smithi046 running (7m) 60s ago 7m 7570k - 18.2.2-1260-g5403a021 267a473d505d 6eec9c927d3f 2024-04-27T04:02:52.225 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-exporter.smithi073 smithi073 running (6m) 65s ago 6m 7478k - 18.2.2-1260-g5403a021 267a473d505d d101eef4888d 2024-04-27T04:02:52.225 INFO:teuthology.orchestra.run.smithi046.stdout:crash.smithi046 smithi046 running (7m) 60s ago 7m 6660k - 18.2.2-1260-g5403a021 267a473d505d 338c449faabd 2024-04-27T04:02:52.225 INFO:teuthology.orchestra.run.smithi046.stdout:crash.smithi073 smithi073 running (6m) 65s ago 6m 6660k - 18.2.2-1260-g5403a021 267a473d505d be1aaee1c655 2024-04-27T04:02:52.225 INFO:teuthology.orchestra.run.smithi046.stdout:grafana.smithi046 smithi046 *:3000 running (5m) 60s ago 6m 84.6M - 9.4.7 954c08fa6188 a834a3988304 2024-04-27T04:02:52.225 INFO:teuthology.orchestra.run.smithi046.stdout:mds.cephfs.smithi046.mycakd smithi046 running (79s) 60s ago 78s 16.4M - 18.2.2-1260-g5403a021 267a473d505d 74d9a0b351ac 2024-04-27T04:02:52.225 INFO:teuthology.orchestra.run.smithi046.stdout:mds.cephfs.smithi046.qgmaql smithi046 running (86s) 60s ago 86s 12.9M - 18.2.2-1260-g5403a021 267a473d505d 741df55feeaf 2024-04-27T04:02:52.225 INFO:teuthology.orchestra.run.smithi046.stdout:mds.cephfs.smithi073.yryono smithi073 running (76s) 65s ago 76s 16.5M - 18.2.2-1260-g5403a021 267a473d505d 40d61213120b 2024-04-27T04:02:52.225 INFO:teuthology.orchestra.run.smithi046.stdout:mds.cephfs.smithi073.zkoypj smithi073 running (83s) 65s ago 83s 18.5M - 18.2.2-1260-g5403a021 267a473d505d 5812a431fb59 2024-04-27T04:02:52.225 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.smithi046.qrfdvr smithi046 *:9283,8765,8443 running (8m) 60s ago 8m 498M - 18.2.2-1260-g5403a021 267a473d505d ba1d99ba4a26 2024-04-27T04:02:52.225 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.smithi073.awyllt smithi073 *:8443,9283,8765 running (5m) 65s ago 5m 451M - 18.2.2-1260-g5403a021 267a473d505d 06769682a51f 2024-04-27T04:02:52.225 INFO:teuthology.orchestra.run.smithi046.stdout:mon.smithi046 smithi046 running (8m) 60s ago 8m 44.1M 2048M 18.2.2-1260-g5403a021 267a473d505d 72fac097effc 2024-04-27T04:02:52.225 INFO:teuthology.orchestra.run.smithi046.stdout:mon.smithi073 smithi073 running (5m) 65s ago 5m 35.2M 2048M 18.2.2-1260-g5403a021 267a473d505d 77769d199765 2024-04-27T04:02:52.225 INFO:teuthology.orchestra.run.smithi046.stdout:node-exporter.smithi046 smithi046 *:9100 running (7m) 60s ago 7m 16.3M - 1.5.0 0da6a335fe13 02c01746eff1 2024-04-27T04:02:52.225 INFO:teuthology.orchestra.run.smithi046.stdout:node-exporter.smithi073 smithi073 *:9100 running (6m) 65s ago 6m 15.2M - 1.5.0 0da6a335fe13 377ef4267056 2024-04-27T04:02:52.226 INFO:teuthology.orchestra.run.smithi046.stdout:osd.0 smithi046 running (5m) 60s ago 5m 57.4M 3948M 18.2.2-1260-g5403a021 267a473d505d 563c995c0df6 2024-04-27T04:02:52.226 INFO:teuthology.orchestra.run.smithi046.stdout:osd.1 smithi046 running (4m) 60s ago 4m 48.7M 3948M 18.2.2-1260-g5403a021 267a473d505d 5673c75e53bb 2024-04-27T04:02:52.226 INFO:teuthology.orchestra.run.smithi046.stdout:osd.2 smithi046 running (4m) 60s ago 4m 46.7M 3948M 18.2.2-1260-g5403a021 267a473d505d 9787b5012842 2024-04-27T04:02:52.226 INFO:teuthology.orchestra.run.smithi046.stdout:osd.3 smithi073 running (3m) 65s ago 3m 46.3M 4970M 18.2.2-1260-g5403a021 267a473d505d 0c89b358284d 2024-04-27T04:02:52.226 INFO:teuthology.orchestra.run.smithi046.stdout:osd.4 smithi073 running (3m) 65s ago 3m 53.4M 4970M 18.2.2-1260-g5403a021 267a473d505d 26f06c87c31e 2024-04-27T04:02:52.226 INFO:teuthology.orchestra.run.smithi046.stdout:osd.5 smithi073 running (2m) 65s ago 2m 49.6M 4970M 18.2.2-1260-g5403a021 267a473d505d e1ce06e0da9b 2024-04-27T04:02:52.226 INFO:teuthology.orchestra.run.smithi046.stdout:prometheus.smithi046 smithi046 *:9095 running (5m) 60s ago 6m 66.2M - 2.43.0 a07b618ecd1d e1a7d8656893 2024-04-27T04:02:52.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.227+0000 7f84e3d17700 1 -- 172.21.15.46:0/1647864543 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f84c806cb30 msgr2=0x7f84c806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:52.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.227+0000 7f84e3d17700 1 --2- 172.21.15.46:0/1647864543 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f84c806cb30 0x7f84c806eff0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00b760 tx=0x7f84cc009f70 comp rx=0 tx=0).stop 2024-04-27T04:02:52.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.227+0000 7f84e3d17700 1 -- 172.21.15.46:0/1647864543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84dc1017a0 msgr2=0x7f84dc196600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:52.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.227+0000 7f84e3d17700 1 --2- 172.21.15.46:0/1647864543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84dc1017a0 0x7f84dc196600 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009cb0 tx=0x7f84d800b2c0 comp rx=0 tx=0).stop 2024-04-27T04:02:52.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.227+0000 7f84e3d17700 1 -- 172.21.15.46:0/1647864543 shutdown_connections 2024-04-27T04:02:52.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.227+0000 7f84e3d17700 1 --2- 172.21.15.46:0/1647864543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84dc1017a0 0x7f84dc196600 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:52.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.227+0000 7f84e3d17700 1 --2- 172.21.15.46:0/1647864543 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f84c806cb30 0x7f84c806eff0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:52.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.227+0000 7f84e3d17700 1 --2- 172.21.15.46:0/1647864543 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84dc1040d0 0x7f84dc196b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:52.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.227+0000 7f84e3d17700 1 -- 172.21.15.46:0/1647864543 >> 172.21.15.46:0/1647864543 conn(0x7f84dc0fb110 msgr2=0x7f84dc0fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:52.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.227+0000 7f84e3d17700 1 -- 172.21.15.46:0/1647864543 shutdown_connections 2024-04-27T04:02:52.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.227+0000 7f84e3d17700 1 -- 172.21.15.46:0/1647864543 wait complete. 2024-04-27T04:02:52.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:51 smithi073 ceph-mon[34160]: pgmap v224: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-04-27T04:02:52.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:51 smithi073 ceph-mon[34160]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:52.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.390+0000 7f894749e700 1 -- 172.21.15.46:0/1740902966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89400ffb90 msgr2=0x7f89400fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:52.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.390+0000 7f894749e700 1 --2- 172.21.15.46:0/1740902966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89400ffb90 0x7f89400fffb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f893c009910 tx=0x7f893c009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:52.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.390+0000 7f894749e700 1 -- 172.21.15.46:0/1740902966 shutdown_connections 2024-04-27T04:02:52.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.390+0000 7f894749e700 1 --2- 172.21.15.46:0/1740902966 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8940100d90 0x7f89401011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:52.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.390+0000 7f894749e700 1 --2- 172.21.15.46:0/1740902966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89400ffb90 0x7f89400fffb0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:52.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.390+0000 7f894749e700 1 -- 172.21.15.46:0/1740902966 >> 172.21.15.46:0/1740902966 conn(0x7f89400fb110 msgr2=0x7f89400fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:52.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.390+0000 7f894749e700 1 -- 172.21.15.46:0/1740902966 shutdown_connections 2024-04-27T04:02:52.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.391+0000 7f894749e700 1 -- 172.21.15.46:0/1740902966 wait complete. 2024-04-27T04:02:52.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.391+0000 7f894749e700 1 Processor -- start 2024-04-27T04:02:52.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.391+0000 7f894749e700 1 -- start start 2024-04-27T04:02:52.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.391+0000 7f894749e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f89400ffb90 0x7f8940196740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:52.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.391+0000 7f894749e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940100d90 0x7f8940196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:52.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.391+0000 7f894749e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89401972a0 con 0x7f8940100d90 2024-04-27T04:02:52.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.391+0000 7f894749e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89401973e0 con 0x7f89400ffb90 2024-04-27T04:02:52.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.392+0000 7f8944a39700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940100d90 0x7f8940196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:52.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.392+0000 7f894523a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f89400ffb90 0x7f8940196740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:52.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.392+0000 7f8944a39700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940100d90 0x7f8940196c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58382/0 (socket says 172.21.15.46:58382) 2024-04-27T04:02:52.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.392+0000 7f8944a39700 1 -- 172.21.15.46:0/575317337 learned_addr learned my addr 172.21.15.46:0/575317337 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:52.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.392+0000 7f8944a39700 1 -- 172.21.15.46:0/575317337 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f89400ffb90 msgr2=0x7f8940196740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:52.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.392+0000 7f8944a39700 1 --2- 172.21.15.46:0/575317337 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f89400ffb90 0x7f8940196740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:52.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.392+0000 7f8944a39700 1 -- 172.21.15.46:0/575317337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893c0095f0 con 0x7f8940100d90 2024-04-27T04:02:52.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.392+0000 7f894523a700 1 --2- 172.21.15.46:0/575317337 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f89400ffb90 0x7f8940196740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T04:02:52.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.392+0000 7f8944a39700 1 --2- 172.21.15.46:0/575317337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940100d90 0x7f8940196c80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f893000e920 tx=0x7f893000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:52.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.392+0000 7f89367fc700 1 -- 172.21.15.46:0/575317337 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f893000c9d0 con 0x7f8940100d90 2024-04-27T04:02:52.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.392+0000 7f89367fc700 1 -- 172.21.15.46:0/575317337 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8930010430 con 0x7f8940100d90 2024-04-27T04:02:52.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.393+0000 7f894749e700 1 -- 172.21.15.46:0/575317337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f894019bdf0 con 0x7f8940100d90 2024-04-27T04:02:52.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.393+0000 7f89367fc700 1 -- 172.21.15.46:0/575317337 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8930018630 con 0x7f8940100d90 2024-04-27T04:02:52.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.393+0000 7f894749e700 1 -- 172.21.15.46:0/575317337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f894019c310 con 0x7f8940100d90 2024-04-27T04:02:52.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.394+0000 7f894749e700 1 -- 172.21.15.46:0/575317337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f894004ebb0 con 0x7f8940100d90 2024-04-27T04:02:52.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.394+0000 7f89367fc700 1 -- 172.21.15.46:0/575317337 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f89300105a0 con 0x7f8940100d90 2024-04-27T04:02:52.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.395+0000 7f89367fc700 1 --2- 172.21.15.46:0/575317337 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f892c06cce0 0x7f892c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:52.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.395+0000 7f89367fc700 1 -- 172.21.15.46:0/575317337 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f8930014070 con 0x7f8940100d90 2024-04-27T04:02:52.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.395+0000 7f894523a700 1 --2- 172.21.15.46:0/575317337 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f892c06cce0 0x7f892c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:52.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.396+0000 7f894523a700 1 --2- 172.21.15.46:0/575317337 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f892c06cce0 0x7f892c06f1a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f893c006010 tx=0x7f893c009f70 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:52.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.403+0000 7f89367fc700 1 -- 172.21.15.46:0/575317337 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f8930052c50 con 0x7f8940100d90 2024-04-27T04:02:52.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.792+0000 7f894749e700 1 -- 172.21.15.46:0/575317337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f89401086f0 con 0x7f8940100d90 2024-04-27T04:02:52.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.793+0000 7f89367fc700 1 -- 172.21.15.46:0/575317337 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f893005a340 con 0x7f8940100d90 2024-04-27T04:02:52.793 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-27T04:02:52.793 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": { 2024-04-27T04:02:52.794 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)": 2 2024-04-27T04:02:52.794 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-27T04:02:52.794 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": { 2024-04-27T04:02:52.794 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)": 2 2024-04-27T04:02:52.794 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-27T04:02:52.794 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": { 2024-04-27T04:02:52.794 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)": 6 2024-04-27T04:02:52.794 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-27T04:02:52.794 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": { 2024-04-27T04:02:52.794 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)": 4 2024-04-27T04:02:52.794 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-27T04:02:52.795 INFO:teuthology.orchestra.run.smithi046.stdout: "overall": { 2024-04-27T04:02:52.795 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)": 14 2024-04-27T04:02:52.795 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-27T04:02:52.795 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-27T04:02:52.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.795+0000 7f894749e700 1 -- 172.21.15.46:0/575317337 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f892c06cce0 msgr2=0x7f892c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:52.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.795+0000 7f894749e700 1 --2- 172.21.15.46:0/575317337 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f892c06cce0 0x7f892c06f1a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f893c006010 tx=0x7f893c009f70 comp rx=0 tx=0).stop 2024-04-27T04:02:52.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.795+0000 7f894749e700 1 -- 172.21.15.46:0/575317337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940100d90 msgr2=0x7f8940196c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:52.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.795+0000 7f894749e700 1 --2- 172.21.15.46:0/575317337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940100d90 0x7f8940196c80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f893000e920 tx=0x7f893000ec30 comp rx=0 tx=0).stop 2024-04-27T04:02:52.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.795+0000 7f894749e700 1 -- 172.21.15.46:0/575317337 shutdown_connections 2024-04-27T04:02:52.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.795+0000 7f894749e700 1 --2- 172.21.15.46:0/575317337 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f892c06cce0 0x7f892c06f1a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:52.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.795+0000 7f894749e700 1 --2- 172.21.15.46:0/575317337 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f89400ffb90 0x7f8940196740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:52.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.795+0000 7f894749e700 1 --2- 172.21.15.46:0/575317337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940100d90 0x7f8940196c80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:52.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.795+0000 7f894749e700 1 -- 172.21.15.46:0/575317337 >> 172.21.15.46:0/575317337 conn(0x7f89400fb110 msgr2=0x7f8940068630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:52.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.795+0000 7f894749e700 1 -- 172.21.15.46:0/575317337 shutdown_connections 2024-04-27T04:02:52.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.795+0000 7f894749e700 1 -- 172.21.15.46:0/575317337 wait complete. 2024-04-27T04:02:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.931+0000 7f8eb4d53700 1 -- 172.21.15.46:0/2558732191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb00ffe50 msgr2=0x7f8eb01002d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.931+0000 7f8eb4d53700 1 --2- 172.21.15.46:0/2558732191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb00ffe50 0x7f8eb01002d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009910 tx=0x7f8ea4009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.931+0000 7f8eb4d53700 1 -- 172.21.15.46:0/2558732191 shutdown_connections 2024-04-27T04:02:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.931+0000 7f8eb4d53700 1 --2- 172.21.15.46:0/2558732191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb00ffe50 0x7f8eb01002d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.931+0000 7f8eb4d53700 1 --2- 172.21.15.46:0/2558732191 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8eb00ff4f0 0x7f8eb00ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.931+0000 7f8eb4d53700 1 -- 172.21.15.46:0/2558732191 >> 172.21.15.46:0/2558732191 conn(0x7f8eb00fb0b0 msgr2=0x7f8eb00fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.931+0000 7f8eb4d53700 1 -- 172.21.15.46:0/2558732191 shutdown_connections 2024-04-27T04:02:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.931+0000 7f8eb4d53700 1 -- 172.21.15.46:0/2558732191 wait complete. 2024-04-27T04:02:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.932+0000 7f8eb4d53700 1 Processor -- start 2024-04-27T04:02:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.932+0000 7f8eb4d53700 1 -- start start 2024-04-27T04:02:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.933+0000 7f8eb4d53700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8eb00ff4f0 0x7f8eb0104fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.933+0000 7f8eb4d53700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb00ffe50 0x7f8eb0105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.933+0000 7f8eb4d53700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb0101b60 con 0x7f8eb00ffe50 2024-04-27T04:02:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.933+0000 7f8eb4d53700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb0101ca0 con 0x7f8eb00ff4f0 2024-04-27T04:02:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.933+0000 7f8eadd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb00ffe50 0x7f8eb0105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.933+0000 7f8eadd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb00ffe50 0x7f8eb0105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58406/0 (socket says 172.21.15.46:58406) 2024-04-27T04:02:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.933+0000 7f8eadd9b700 1 -- 172.21.15.46:0/648800006 learned_addr learned my addr 172.21.15.46:0/648800006 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.933+0000 7f8eae59c700 1 --2- 172.21.15.46:0/648800006 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8eb00ff4f0 0x7f8eb0104fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.933+0000 7f8eadd9b700 1 -- 172.21.15.46:0/648800006 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8eb00ff4f0 msgr2=0x7f8eb0104fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.933+0000 7f8eadd9b700 1 --2- 172.21.15.46:0/648800006 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8eb00ff4f0 0x7f8eb0104fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.933+0000 7f8eadd9b700 1 -- 172.21.15.46:0/648800006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea40095f0 con 0x7f8eb00ffe50 2024-04-27T04:02:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.933+0000 7f8eadd9b700 1 --2- 172.21.15.46:0/648800006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb00ffe50 0x7f8eb0105510 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4005b40 tx=0x7f8ea400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.934+0000 7f8e9f7fe700 1 -- 172.21.15.46:0/648800006 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ea401d070 con 0x7f8eb00ffe50 2024-04-27T04:02:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.934+0000 7f8e9f7fe700 1 -- 172.21.15.46:0/648800006 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ea40050d0 con 0x7f8eb00ffe50 2024-04-27T04:02:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.934+0000 7f8e9f7fe700 1 -- 172.21.15.46:0/648800006 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ea4022470 con 0x7f8eb00ffe50 2024-04-27T04:02:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.934+0000 7f8eb4d53700 1 -- 172.21.15.46:0/648800006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb0101f20 con 0x7f8eb00ffe50 2024-04-27T04:02:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.934+0000 7f8eb4d53700 1 -- 172.21.15.46:0/648800006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb0102490 con 0x7f8eb00ffe50 2024-04-27T04:02:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.935+0000 7f8eb4d53700 1 -- 172.21.15.46:0/648800006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb0192990 con 0x7f8eb00ffe50 2024-04-27T04:02:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.936+0000 7f8e9f7fe700 1 -- 172.21.15.46:0/648800006 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8ea40037f0 con 0x7f8eb00ffe50 2024-04-27T04:02:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.936+0000 7f8e9f7fe700 1 --2- 172.21.15.46:0/648800006 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8e9806ccf0 0x7f8e9806f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.937+0000 7f8e9f7fe700 1 -- 172.21.15.46:0/648800006 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f8ea408d780 con 0x7f8eb00ffe50 2024-04-27T04:02:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.937+0000 7f8eae59c700 1 --2- 172.21.15.46:0/648800006 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8e9806ccf0 0x7f8e9806f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:52.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.937+0000 7f8eae59c700 1 --2- 172.21.15.46:0/648800006 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8e9806ccf0 0x7f8e9806f1b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0009c60 tx=0x7f8ea0009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:52.941+0000 7f8e9f7fe700 1 -- 172.21.15.46:0/648800006 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f8ea405b600 con 0x7f8eb00ffe50 2024-04-27T04:02:53.144 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:52 smithi073 ceph-mon[34160]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:53.144 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:52 smithi073 ceph-mon[34160]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:53.144 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:52 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/575317337' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T04:02:53.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:52 smithi046 ceph-mon[26899]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:53.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:52 smithi046 ceph-mon[26899]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:53.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:52 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/575317337' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T04:02:53.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.294+0000 7f8eb4d53700 1 -- 172.21.15.46:0/648800006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8eb010bc00 con 0x7f8eb00ffe50 2024-04-27T04:02:53.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.295+0000 7f8e9f7fe700 1 -- 172.21.15.46:0/648800006 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1865 (secure 0 0 0) 0x7f8ea405b190 con 0x7f8eb00ffe50 2024-04-27T04:02:53.296 INFO:teuthology.orchestra.run.smithi046.stdout:e19 2024-04-27T04:02:53.296 INFO:teuthology.orchestra.run.smithi046.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T04:02:53.296 INFO:teuthology.orchestra.run.smithi046.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T04:02:53.296 INFO:teuthology.orchestra.run.smithi046.stdout:legacy client fscid: 1 2024-04-27T04:02:53.296 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:02:53.296 INFO:teuthology.orchestra.run.smithi046.stdout:Filesystem 'cephfs' (1) 2024-04-27T04:02:53.296 INFO:teuthology.orchestra.run.smithi046.stdout:fs_name cephfs 2024-04-27T04:02:53.296 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 19 2024-04-27T04:02:53.296 INFO:teuthology.orchestra.run.smithi046.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-27T04:02:53.296 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-04-27T04:01:21.676811+0000 2024-04-27T04:02:53.297 INFO:teuthology.orchestra.run.smithi046.stdout:modified 2024-04-27T04:01:42.132288+0000 2024-04-27T04:02:53.297 INFO:teuthology.orchestra.run.smithi046.stdout:tableserver 0 2024-04-27T04:02:53.297 INFO:teuthology.orchestra.run.smithi046.stdout:root 0 2024-04-27T04:02:53.297 INFO:teuthology.orchestra.run.smithi046.stdout:session_timeout 60 2024-04-27T04:02:53.297 INFO:teuthology.orchestra.run.smithi046.stdout:session_autoclose 300 2024-04-27T04:02:53.297 INFO:teuthology.orchestra.run.smithi046.stdout:max_file_size 1099511627776 2024-04-27T04:02:53.297 INFO:teuthology.orchestra.run.smithi046.stdout:max_xattr_size 65536 2024-04-27T04:02:53.297 INFO:teuthology.orchestra.run.smithi046.stdout:required_client_features {} 2024-04-27T04:02:53.297 INFO:teuthology.orchestra.run.smithi046.stdout:last_failure 0 2024-04-27T04:02:53.297 INFO:teuthology.orchestra.run.smithi046.stdout:last_failure_osd_epoch 46 2024-04-27T04:02:53.297 INFO:teuthology.orchestra.run.smithi046.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T04:02:53.297 INFO:teuthology.orchestra.run.smithi046.stdout:max_mds 2 2024-04-27T04:02:53.297 INFO:teuthology.orchestra.run.smithi046.stdout:in 0,1 2024-04-27T04:02:53.298 INFO:teuthology.orchestra.run.smithi046.stdout:up {0=14488,1=24285} 2024-04-27T04:02:53.298 INFO:teuthology.orchestra.run.smithi046.stdout:failed 2024-04-27T04:02:53.298 INFO:teuthology.orchestra.run.smithi046.stdout:damaged 2024-04-27T04:02:53.298 INFO:teuthology.orchestra.run.smithi046.stdout:stopped 2024-04-27T04:02:53.298 INFO:teuthology.orchestra.run.smithi046.stdout:data_pools [3] 2024-04-27T04:02:53.298 INFO:teuthology.orchestra.run.smithi046.stdout:metadata_pool 2 2024-04-27T04:02:53.298 INFO:teuthology.orchestra.run.smithi046.stdout:inline_data enabled 2024-04-27T04:02:53.298 INFO:teuthology.orchestra.run.smithi046.stdout:balancer 2024-04-27T04:02:53.298 INFO:teuthology.orchestra.run.smithi046.stdout:bal_rank_mask -1 2024-04-27T04:02:53.298 INFO:teuthology.orchestra.run.smithi046.stdout:standby_count_wanted 1 2024-04-27T04:02:53.298 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.cephfs.smithi046.mycakd{0:14488} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.46:6828/3081610305,v1:172.21.15.46:6829/3081610305] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T04:02:53.298 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.cephfs.smithi046.qgmaql{0:24303} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.46:6826/2653255940,v1:172.21.15.46:6827/2653255940] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T04:02:53.298 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.cephfs.smithi073.zkoypj{1:24285} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.73:6824/3397960510,v1:172.21.15.73:6825/3397960510] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T04:02:53.299 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.cephfs.smithi073.yryono{1:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.73:6826/565262827,v1:172.21.15.73:6827/565262827] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T04:02:53.299 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:02:53.299 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:02:53.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.298+0000 7f8eb4d53700 1 -- 172.21.15.46:0/648800006 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8e9806ccf0 msgr2=0x7f8e9806f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:53.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.298+0000 7f8eb4d53700 1 --2- 172.21.15.46:0/648800006 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8e9806ccf0 0x7f8e9806f1b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0009c60 tx=0x7f8ea0009260 comp rx=0 tx=0).stop 2024-04-27T04:02:53.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.298+0000 7f8eb4d53700 1 -- 172.21.15.46:0/648800006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb00ffe50 msgr2=0x7f8eb0105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:53.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.298+0000 7f8eb4d53700 1 --2- 172.21.15.46:0/648800006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb00ffe50 0x7f8eb0105510 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4005b40 tx=0x7f8ea400f690 comp rx=0 tx=0).stop 2024-04-27T04:02:53.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.298+0000 7f8eb4d53700 1 -- 172.21.15.46:0/648800006 shutdown_connections 2024-04-27T04:02:53.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.298+0000 7f8eb4d53700 1 --2- 172.21.15.46:0/648800006 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f8e9806ccf0 0x7f8e9806f1b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:53.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.298+0000 7f8eb4d53700 1 --2- 172.21.15.46:0/648800006 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8eb00ff4f0 0x7f8eb0104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:53.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.298+0000 7f8eb4d53700 1 --2- 172.21.15.46:0/648800006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb00ffe50 0x7f8eb0105510 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:53.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.298+0000 7f8eb4d53700 1 -- 172.21.15.46:0/648800006 >> 172.21.15.46:0/648800006 conn(0x7f8eb00fb0b0 msgr2=0x7f8eb0107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:53.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.298+0000 7f8eb4d53700 1 -- 172.21.15.46:0/648800006 shutdown_connections 2024-04-27T04:02:53.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.298+0000 7f8eb4d53700 1 -- 172.21.15.46:0/648800006 wait complete. 2024-04-27T04:02:53.300 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 19 2024-04-27T04:02:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.463+0000 7f6fc7950700 1 -- 172.21.15.46:0/2200961385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc0101e30 msgr2=0x7f6fc0102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.463+0000 7f6fc7950700 1 --2- 172.21.15.46:0/2200961385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc0101e30 0x7f6fc0102290 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0009910 tx=0x7f6fb0009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.463+0000 7f6fc7950700 1 -- 172.21.15.46:0/2200961385 shutdown_connections 2024-04-27T04:02:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.463+0000 7f6fc7950700 1 --2- 172.21.15.46:0/2200961385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc0101e30 0x7f6fc0102290 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.463+0000 7f6fc7950700 1 --2- 172.21.15.46:0/2200961385 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6fc0100c30 0x7f6fc0101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.463+0000 7f6fc7950700 1 -- 172.21.15.46:0/2200961385 >> 172.21.15.46:0/2200961385 conn(0x7f6fc00fc1d0 msgr2=0x7f6fc00fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.464+0000 7f6fc7950700 1 -- 172.21.15.46:0/2200961385 shutdown_connections 2024-04-27T04:02:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.464+0000 7f6fc7950700 1 -- 172.21.15.46:0/2200961385 wait complete. 2024-04-27T04:02:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.464+0000 7f6fc7950700 1 Processor -- start 2024-04-27T04:02:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.465+0000 7f6fc7950700 1 -- start start 2024-04-27T04:02:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.465+0000 7f6fc7950700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6fc0100c30 0x7f6fc0192930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.465+0000 7f6fc7950700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc0101e30 0x7f6fc0192e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.465+0000 7f6fc7950700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc018f470 con 0x7f6fc0101e30 2024-04-27T04:02:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.465+0000 7f6fc7950700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc018f5b0 con 0x7f6fc0100c30 2024-04-27T04:02:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.465+0000 7f6fc56ec700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6fc0100c30 0x7f6fc0192930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.465+0000 7f6fc56ec700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6fc0100c30 0x7f6fc0192930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.46:53568/0 (socket says 172.21.15.46:53568) 2024-04-27T04:02:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.465+0000 7f6fc4eeb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc0101e30 0x7f6fc0192e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.465+0000 7f6fc56ec700 1 -- 172.21.15.46:0/4119130561 learned_addr learned my addr 172.21.15.46:0/4119130561 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.465+0000 7f6fc56ec700 1 -- 172.21.15.46:0/4119130561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc0101e30 msgr2=0x7f6fc0192e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.465+0000 7f6fc56ec700 1 --2- 172.21.15.46:0/4119130561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc0101e30 0x7f6fc0192e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.465+0000 7f6fc56ec700 1 -- 172.21.15.46:0/4119130561 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb00095f0 con 0x7f6fc0100c30 2024-04-27T04:02:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.466+0000 7f6fc56ec700 1 --2- 172.21.15.46:0/4119130561 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6fc0100c30 0x7f6fc0192930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00e8f0 tx=0x7f6fbc00ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.466+0000 7f6fb67fc700 1 -- 172.21.15.46:0/4119130561 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fbc009790 con 0x7f6fc0100c30 2024-04-27T04:02:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.466+0000 7f6fc7950700 1 -- 172.21.15.46:0/4119130561 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc018f890 con 0x7f6fc0100c30 2024-04-27T04:02:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.466+0000 7f6fc7950700 1 -- 172.21.15.46:0/4119130561 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc018fe60 con 0x7f6fc0100c30 2024-04-27T04:02:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.466+0000 7f6fb67fc700 1 -- 172.21.15.46:0/4119130561 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fbc00c810 con 0x7f6fc0100c30 2024-04-27T04:02:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.466+0000 7f6fb67fc700 1 -- 172.21.15.46:0/4119130561 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fbc003c80 con 0x7f6fc0100c30 2024-04-27T04:02:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.467+0000 7f6fc7950700 1 -- 172.21.15.46:0/4119130561 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa40052f0 con 0x7f6fc0100c30 2024-04-27T04:02:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.468+0000 7f6fb67fc700 1 -- 172.21.15.46:0/4119130561 <== mon.1 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6fbc00c980 con 0x7f6fc0100c30 2024-04-27T04:02:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.469+0000 7f6fb67fc700 1 --2- 172.21.15.46:0/4119130561 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f6fac06cce0 0x7f6fac06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:53.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.469+0000 7f6fb67fc700 1 -- 172.21.15.46:0/4119130561 <== mon.1 v2:172.21.15.73:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f6fbc014070 con 0x7f6fc0100c30 2024-04-27T04:02:53.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.469+0000 7f6fc4eeb700 1 --2- 172.21.15.46:0/4119130561 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f6fac06cce0 0x7f6fac06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:53.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.470+0000 7f6fc4eeb700 1 --2- 172.21.15.46:0/4119130561 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f6fac06cce0 0x7f6fac06f1a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6fb000b800 tx=0x7f6fb0009f70 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:53.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.478+0000 7f6fb67fc700 1 -- 172.21.15.46:0/4119130561 <== mon.1 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f6fbc05a2e0 con 0x7f6fc0100c30 2024-04-27T04:02:53.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.723+0000 7f6fc7950700 1 -- 172.21.15.46:0/4119130561 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fa4000bc0 con 0x7f6fac06cce0 2024-04-27T04:02:53.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.725+0000 7f6fb67fc700 1 -- 172.21.15.46:0/4119130561 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f6fa4000bc0 con 0x7f6fac06cce0 2024-04-27T04:02:53.725 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-27T04:02:53.725 INFO:teuthology.orchestra.run.smithi046.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:90a3a9fa9ceb8977f6dad9ccd36fe46018d07e2118199820523a6b35301d9c16", 2024-04-27T04:02:53.725 INFO:teuthology.orchestra.run.smithi046.stdout: "in_progress": true, 2024-04-27T04:02:53.725 INFO:teuthology.orchestra.run.smithi046.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T04:02:53.725 INFO:teuthology.orchestra.run.smithi046.stdout: "services_complete": [], 2024-04-27T04:02:53.725 INFO:teuthology.orchestra.run.smithi046.stdout: "progress": "0/23 daemons upgraded", 2024-04-27T04:02:53.726 INFO:teuthology.orchestra.run.smithi046.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:90a3a9fa9ceb8977f6dad9ccd36fe46018d07e2118199820523a6b35301d9c16 image on host smithi073", 2024-04-27T04:02:53.726 INFO:teuthology.orchestra.run.smithi046.stdout: "is_paused": false 2024-04-27T04:02:53.726 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-27T04:02:53.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.728+0000 7f6fc7950700 1 -- 172.21.15.46:0/4119130561 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f6fac06cce0 msgr2=0x7f6fac06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:53.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.728+0000 7f6fc7950700 1 --2- 172.21.15.46:0/4119130561 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f6fac06cce0 0x7f6fac06f1a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6fb000b800 tx=0x7f6fb0009f70 comp rx=0 tx=0).stop 2024-04-27T04:02:53.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.728+0000 7f6fc7950700 1 -- 172.21.15.46:0/4119130561 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6fc0100c30 msgr2=0x7f6fc0192930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:53.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.728+0000 7f6fc7950700 1 --2- 172.21.15.46:0/4119130561 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6fc0100c30 0x7f6fc0192930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00e8f0 tx=0x7f6fbc00ec00 comp rx=0 tx=0).stop 2024-04-27T04:02:53.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.728+0000 7f6fc7950700 1 -- 172.21.15.46:0/4119130561 shutdown_connections 2024-04-27T04:02:53.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.728+0000 7f6fc7950700 1 --2- 172.21.15.46:0/4119130561 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f6fac06cce0 0x7f6fac06f1a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:53.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.728+0000 7f6fc7950700 1 --2- 172.21.15.46:0/4119130561 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6fc0100c30 0x7f6fc0192930 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:53.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.728+0000 7f6fc7950700 1 --2- 172.21.15.46:0/4119130561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc0101e30 0x7f6fc0192e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:53.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.728+0000 7f6fc7950700 1 -- 172.21.15.46:0/4119130561 >> 172.21.15.46:0/4119130561 conn(0x7f6fc00fc1d0 msgr2=0x7f6fc0073440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:53.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.728+0000 7f6fc7950700 1 -- 172.21.15.46:0/4119130561 shutdown_connections 2024-04-27T04:02:53.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.728+0000 7f6fc7950700 1 -- 172.21.15.46:0/4119130561 wait complete. 2024-04-27T04:02:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.885+0000 7ff4811c7700 1 -- 172.21.15.46:0/3923276115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff47c100c10 msgr2=0x7ff47c101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.885+0000 7ff4811c7700 1 --2- 172.21.15.46:0/3923276115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff47c100c10 0x7ff47c101030 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009910 tx=0x7ff46c009c20 comp rx=0 tx=0).stop 2024-04-27T04:02:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.885+0000 7ff4811c7700 1 -- 172.21.15.46:0/3923276115 shutdown_connections 2024-04-27T04:02:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.885+0000 7ff4811c7700 1 --2- 172.21.15.46:0/3923276115 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff47c101e10 0x7ff47c102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.885+0000 7ff4811c7700 1 --2- 172.21.15.46:0/3923276115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff47c100c10 0x7ff47c101030 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.885+0000 7ff4811c7700 1 -- 172.21.15.46:0/3923276115 >> 172.21.15.46:0/3923276115 conn(0x7ff47c0fc1f0 msgr2=0x7ff47c0fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.885+0000 7ff4811c7700 1 -- 172.21.15.46:0/3923276115 shutdown_connections 2024-04-27T04:02:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.885+0000 7ff4811c7700 1 -- 172.21.15.46:0/3923276115 wait complete. 2024-04-27T04:02:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.886+0000 7ff4811c7700 1 Processor -- start 2024-04-27T04:02:53.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.886+0000 7ff4811c7700 1 -- start start 2024-04-27T04:02:53.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.886+0000 7ff4811c7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff47c100c10 0x7ff47c196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:53.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.886+0000 7ff4811c7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff47c101e10 0x7ff47c196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:53.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.886+0000 7ff4811c7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c197190 con 0x7ff47c101e10 2024-04-27T04:02:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.886+0000 7ff4811c7700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c1972d0 con 0x7ff47c100c10 2024-04-27T04:02:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.887+0000 7ff47ad9d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff47c100c10 0x7ff47c196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.887+0000 7ff47a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff47c101e10 0x7ff47c196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.887+0000 7ff47a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff47c101e10 0x7ff47c196b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58450/0 (socket says 172.21.15.46:58450) 2024-04-27T04:02:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.887+0000 7ff47a59c700 1 -- 172.21.15.46:0/4155287637 learned_addr learned my addr 172.21.15.46:0/4155287637 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:02:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.887+0000 7ff47a59c700 1 -- 172.21.15.46:0/4155287637 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff47c100c10 msgr2=0x7ff47c196630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.887+0000 7ff47a59c700 1 --2- 172.21.15.46:0/4155287637 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff47c100c10 0x7ff47c196630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.887+0000 7ff47a59c700 1 -- 172.21.15.46:0/4155287637 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46c0095f0 con 0x7ff47c101e10 2024-04-27T04:02:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.887+0000 7ff47ad9d700 1 --2- 172.21.15.46:0/4155287637 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff47c100c10 0x7ff47c196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T04:02:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.887+0000 7ff47a59c700 1 --2- 172.21.15.46:0/4155287637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff47c101e10 0x7ff47c196b70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff47000e920 tx=0x7ff47000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.888+0000 7ff463fff700 1 -- 172.21.15.46:0/4155287637 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff47000c9d0 con 0x7ff47c101e10 2024-04-27T04:02:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.888+0000 7ff463fff700 1 -- 172.21.15.46:0/4155287637 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff470010430 con 0x7ff47c101e10 2024-04-27T04:02:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.888+0000 7ff4811c7700 1 -- 172.21.15.46:0/4155287637 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff47c073170 con 0x7ff47c101e10 2024-04-27T04:02:53.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.888+0000 7ff463fff700 1 -- 172.21.15.46:0/4155287637 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff470018630 con 0x7ff47c101e10 2024-04-27T04:02:53.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.888+0000 7ff4811c7700 1 -- 172.21.15.46:0/4155287637 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff47c073740 con 0x7ff47c101e10 2024-04-27T04:02:53.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.889+0000 7ff4811c7700 1 -- 172.21.15.46:0/4155287637 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff47c04ebb0 con 0x7ff47c101e10 2024-04-27T04:02:53.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.890+0000 7ff463fff700 1 -- 172.21.15.46:0/4155287637 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff4700105a0 con 0x7ff47c101e10 2024-04-27T04:02:53.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.890+0000 7ff463fff700 1 --2- 172.21.15.46:0/4155287637 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff46406cd30 0x7ff46406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:02:53.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.890+0000 7ff463fff700 1 -- 172.21.15.46:0/4155287637 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff470014070 con 0x7ff47c101e10 2024-04-27T04:02:53.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.891+0000 7ff47ad9d700 1 --2- 172.21.15.46:0/4155287637 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff46406cd30 0x7ff46406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:02:53.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.891+0000 7ff47ad9d700 1 --2- 172.21.15.46:0/4155287637 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff46406cd30 0x7ff46406f1f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00b800 tx=0x7ff46c009f70 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:02:53.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:53.899+0000 7ff463fff700 1 -- 172.21.15.46:0/4155287637 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7ff470052c50 con 0x7ff47c101e10 2024-04-27T04:02:54.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:53 smithi046 ceph-mon[26899]: pgmap v225: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-27T04:02:54.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:53 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/648800006' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T04:02:54.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:54.247+0000 7ff4811c7700 1 -- 172.21.15.46:0/4155287637 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff47c109770 con 0x7ff47c101e10 2024-04-27T04:02:54.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:54.247+0000 7ff463fff700 1 -- 172.21.15.46:0/4155287637 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7ff47005a320 con 0x7ff47c101e10 2024-04-27T04:02:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-27T04:02:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-27T04:02:54.248 INFO:teuthology.orchestra.run.smithi046.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-27T04:02:54.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:54.249+0000 7ff4811c7700 1 -- 172.21.15.46:0/4155287637 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff46406cd30 msgr2=0x7ff46406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:54.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:54.249+0000 7ff4811c7700 1 --2- 172.21.15.46:0/4155287637 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff46406cd30 0x7ff46406f1f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00b800 tx=0x7ff46c009f70 comp rx=0 tx=0).stop 2024-04-27T04:02:54.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:54.249+0000 7ff4811c7700 1 -- 172.21.15.46:0/4155287637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff47c101e10 msgr2=0x7ff47c196b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:02:54.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:54.249+0000 7ff4811c7700 1 --2- 172.21.15.46:0/4155287637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff47c101e10 0x7ff47c196b70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff47000e920 tx=0x7ff47000ec30 comp rx=0 tx=0).stop 2024-04-27T04:02:54.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:54.250+0000 7ff4811c7700 1 -- 172.21.15.46:0/4155287637 shutdown_connections 2024-04-27T04:02:54.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:54.250+0000 7ff4811c7700 1 --2- 172.21.15.46:0/4155287637 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff46406cd30 0x7ff46406f1f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:54.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:54.250+0000 7ff4811c7700 1 --2- 172.21.15.46:0/4155287637 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff47c100c10 0x7ff47c196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:54.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:54.250+0000 7ff4811c7700 1 --2- 172.21.15.46:0/4155287637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff47c101e10 0x7ff47c196b70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:02:54.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:54.250+0000 7ff4811c7700 1 -- 172.21.15.46:0/4155287637 >> 172.21.15.46:0/4155287637 conn(0x7ff47c0fc1f0 msgr2=0x7ff47c105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:02:54.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:54.250+0000 7ff4811c7700 1 -- 172.21.15.46:0/4155287637 shutdown_connections 2024-04-27T04:02:54.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:02:54.250+0000 7ff4811c7700 1 -- 172.21.15.46:0/4155287637 wait complete. 2024-04-27T04:02:54.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:53 smithi073 ceph-mon[34160]: pgmap v225: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-27T04:02:54.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:53 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/648800006' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T04:02:55.138 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:54 smithi046 ceph-mon[26899]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:55.138 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:54 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/4155287637' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T04:02:55.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:54 smithi073 ceph-mon[34160]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:02:55.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:54 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/4155287637' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T04:02:56.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:55 smithi046 ceph-mon[26899]: pgmap v226: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-27T04:02:56.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:55 smithi073 ceph-mon[34160]: pgmap v226: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-27T04:02:58.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:57 smithi046 ceph-mon[26899]: pgmap v227: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-04-27T04:02:58.235 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:57 smithi073 ceph-mon[34160]: pgmap v227: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-04-27T04:03:00.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:02:59 smithi046 ceph-mon[26899]: pgmap v228: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-04-27T04:03:00.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:02:59 smithi073 ceph-mon[34160]: pgmap v228: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-04-27T04:03:01.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:00 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:03:01.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:00 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:03:02.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:01 smithi046 ceph-mon[26899]: pgmap v229: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-27T04:03:02.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:01 smithi073 ceph-mon[34160]: pgmap v229: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-27T04:03:04.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:03 smithi046 ceph-mon[26899]: pgmap v230: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:03:04.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:03 smithi073 ceph-mon[34160]: pgmap v230: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:03:06.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:05 smithi046 ceph-mon[26899]: pgmap v231: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:03:06.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:05 smithi073 ceph-mon[34160]: pgmap v231: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:03:08.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:07 smithi046 ceph-mon[26899]: pgmap v232: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:03:08.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:07 smithi073 ceph-mon[34160]: pgmap v232: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:03:10.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:09 smithi046 ceph-mon[26899]: pgmap v233: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:03:10.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:09 smithi073 ceph-mon[34160]: pgmap v233: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:03:12.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:11 smithi046 ceph-mon[26899]: pgmap v234: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-27T04:03:12.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:11 smithi073 ceph-mon[34160]: pgmap v234: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-27T04:03:14.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:13 smithi046 ceph-mon[26899]: pgmap v235: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:03:14.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:13 smithi073 ceph-mon[34160]: pgmap v235: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:03:15.365 INFO:tasks.workunit.client.1.smithi073.stderr:Updating files: 87% (10970/12492) Updating files: 88% (10993/12492) Updating files: 89% (11118/12492) Updating files: 90% (11243/12492) Updating files: 91% (11368/12492) Updating files: 92% (11493/12492) Updating files: 93% (11618/12492) Updating files: 94% (11743/12492) Updating files: 95% (11868/12492) Updating files: 96% (11993/12492) Updating files: 97% (12118/12492) Updating files: 98% (12243/12492) Updating files: 99% (12368/12492) Updating files: 100% (12492/12492) Updating files: 100% (12492/12492), done. 2024-04-27T04:03:15.729 INFO:tasks.workunit.client.1.smithi073.stderr:Note: switching to '3067e9f133f954b642fdf21114a79edb58590948'. 2024-04-27T04:03:15.730 INFO:tasks.workunit.client.1.smithi073.stderr: 2024-04-27T04:03:15.730 INFO:tasks.workunit.client.1.smithi073.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-27T04:03:15.730 INFO:tasks.workunit.client.1.smithi073.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-27T04:03:15.730 INFO:tasks.workunit.client.1.smithi073.stderr:state without impacting any branches by switching back to a branch. 2024-04-27T04:03:15.730 INFO:tasks.workunit.client.1.smithi073.stderr: 2024-04-27T04:03:15.730 INFO:tasks.workunit.client.1.smithi073.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-27T04:03:15.730 INFO:tasks.workunit.client.1.smithi073.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-27T04:03:15.730 INFO:tasks.workunit.client.1.smithi073.stderr: 2024-04-27T04:03:15.730 INFO:tasks.workunit.client.1.smithi073.stderr: git switch -c 2024-04-27T04:03:15.730 INFO:tasks.workunit.client.1.smithi073.stderr: 2024-04-27T04:03:15.730 INFO:tasks.workunit.client.1.smithi073.stderr:Or undo this operation with: 2024-04-27T04:03:15.730 INFO:tasks.workunit.client.1.smithi073.stderr: 2024-04-27T04:03:15.731 INFO:tasks.workunit.client.1.smithi073.stderr: git switch - 2024-04-27T04:03:15.731 INFO:tasks.workunit.client.1.smithi073.stderr: 2024-04-27T04:03:15.731 INFO:tasks.workunit.client.1.smithi073.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-27T04:03:15.731 INFO:tasks.workunit.client.1.smithi073.stderr: 2024-04-27T04:03:15.731 INFO:tasks.workunit.client.1.smithi073.stderr:HEAD is now at 3067e9f133f Merge pull request #57105 from zdover23/wip-doc-2024-04-26-mgr-dashboard 2024-04-27T04:03:15.732 DEBUG:teuthology.orchestra.run.smithi073:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-27T04:03:15.812 INFO:tasks.workunit.client.1.smithi073.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-27T04:03:15.814 INFO:tasks.workunit.client.1.smithi073.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-27T04:03:15.814 INFO:tasks.workunit.client.1.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-27T04:03:15.960 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 95% (11870/12492) Updating files: 96% (11993/12492) Updating files: 97% (12118/12492) Updating files: 98% (12243/12492) Updating files: 99% (12368/12492) Updating files: 100% (12492/12492) Updating files: 100% (12492/12492), done. 2024-04-27T04:03:16.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:15 smithi046 ceph-mon[26899]: pgmap v236: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:03:16.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:15 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:03:16.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:15 smithi073 ceph-mon[34160]: pgmap v236: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:03:16.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:15 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:03:16.597 INFO:tasks.workunit.client.1.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-27T04:03:16.639 INFO:tasks.workunit.client.0.smithi046.stderr:Note: switching to '3067e9f133f954b642fdf21114a79edb58590948'. 2024-04-27T04:03:16.640 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-04-27T04:03:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-27T04:03:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-27T04:03:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:state without impacting any branches by switching back to a branch. 2024-04-27T04:03:16.640 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-04-27T04:03:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-27T04:03:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-27T04:03:16.640 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-04-27T04:03:16.640 INFO:tasks.workunit.client.0.smithi046.stderr: git switch -c 2024-04-27T04:03:16.640 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-04-27T04:03:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:Or undo this operation with: 2024-04-27T04:03:16.641 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-04-27T04:03:16.641 INFO:tasks.workunit.client.0.smithi046.stderr: git switch - 2024-04-27T04:03:16.641 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-04-27T04:03:16.641 INFO:tasks.workunit.client.0.smithi046.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-27T04:03:16.641 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-04-27T04:03:16.641 INFO:tasks.workunit.client.0.smithi046.stderr:HEAD is now at 3067e9f133f Merge pull request #57105 from zdover23/wip-doc-2024-04-26-mgr-dashboard 2024-04-27T04:03:16.642 DEBUG:teuthology.orchestra.run.smithi046:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-27T04:03:16.679 INFO:tasks.workunit.client.1.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-27T04:03:16.679 INFO:tasks.workunit.client.0.smithi046.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-27T04:03:16.682 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-27T04:03:16.682 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-27T04:03:16.721 INFO:tasks.workunit.client.1.smithi073.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-27T04:03:16.722 INFO:tasks.workunit.client.1.smithi073.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-27T04:03:16.722 INFO:tasks.workunit.client.1.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-27T04:03:16.764 INFO:tasks.workunit.client.1.smithi073.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-27T04:03:16.767 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-04-27T04:03:16.767 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-27T04:03:16.823 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-04-27T04:03:16.825 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-27T04:03:16.825 DEBUG:teuthology.orchestra.run.smithi073:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3067e9f133f954b642fdf21114a79edb58590948 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-27T04:03:16.886 INFO:tasks.workunit.client.1.smithi073.stderr:+ mkdir -p fsstress 2024-04-27T04:03:16.889 INFO:tasks.workunit.client.1.smithi073.stderr:+ pushd fsstress 2024-04-27T04:03:16.889 INFO:tasks.workunit.client.1.smithi073.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-27T04:03:16.889 INFO:tasks.workunit.client.1.smithi073.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-27T04:03:17.310 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-27T04:03:17.364 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-27T04:03:17.404 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-27T04:03:17.406 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-27T04:03:17.406 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-27T04:03:17.418 INFO:tasks.workunit.client.1.smithi073.stderr:+ tar xzf ltp-full.tgz 2024-04-27T04:03:17.447 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-27T04:03:17.450 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T04:03:17.450 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-27T04:03:17.506 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-04-27T04:03:17.509 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-27T04:03:17.509 DEBUG:teuthology.orchestra.run.smithi046:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3067e9f133f954b642fdf21114a79edb58590948 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-04-27T04:03:17.569 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir -p fsstress 2024-04-27T04:03:17.571 INFO:tasks.workunit.client.0.smithi046.stderr:+ pushd fsstress 2024-04-27T04:03:17.571 INFO:tasks.workunit.client.0.smithi046.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-27T04:03:17.572 INFO:tasks.workunit.client.0.smithi046.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-27T04:03:17.980 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:17 smithi073 ceph-mon[34160]: pgmap v237: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:03:18.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:17 smithi046 ceph-mon[26899]: pgmap v237: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:03:18.383 INFO:tasks.workunit.client.0.smithi046.stderr:+ tar xzf ltp-full.tgz 2024-04-27T04:03:20.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:19 smithi073 ceph-mon[34160]: pgmap v238: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:03:20.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:19 smithi073 ceph-mon[34160]: Upgrade: Updating mgr.smithi073.awyllt 2024-04-27T04:03:20.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:19 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:20.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:19 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi073.awyllt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T04:03:20.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:19 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T04:03:20.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:19 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:03:20.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:19 smithi073 ceph-mon[34160]: Deploying daemon mgr.smithi073.awyllt on smithi073 2024-04-27T04:03:20.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:19 smithi046 ceph-mon[26899]: pgmap v238: 65 pgs: 65 active+clean; 589 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-27T04:03:20.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:19 smithi046 ceph-mon[26899]: Upgrade: Updating mgr.smithi073.awyllt 2024-04-27T04:03:20.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:19 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:20.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:19 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi073.awyllt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T04:03:20.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:19 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T04:03:20.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:19 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:03:20.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:19 smithi046 ceph-mon[26899]: Deploying daemon mgr.smithi073.awyllt on smithi073 2024-04-27T04:03:22.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:21 smithi073 ceph-mon[34160]: pgmap v239: 65 pgs: 65 active+clean; 589 KiB data, 190 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-27T04:03:22.422 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:21 smithi046 ceph-mon[26899]: pgmap v239: 65 pgs: 65 active+clean; 589 KiB data, 190 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-27T04:03:24.261 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:23 smithi073 ceph-mon[34160]: pgmap v240: 65 pgs: 65 active+clean; 7.0 MiB data, 194 MiB used, 536 GiB / 536 GiB avail; 343 KiB/s rd, 548 KiB/s wr, 5 op/s 2024-04-27T04:03:24.370 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:23 smithi046 ceph-mon[26899]: pgmap v240: 65 pgs: 65 active+clean; 7.0 MiB data, 194 MiB used, 536 GiB / 536 GiB avail; 343 KiB/s rd, 548 KiB/s wr, 5 op/s 2024-04-27T04:03:24.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.369+0000 7fb7878e6700 1 -- 172.21.15.46:0/2739419016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb780104050 msgr2=0x7fb780106440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:24.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.369+0000 7fb7878e6700 1 --2- 172.21.15.46:0/2739419016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb780104050 0x7fb780106440 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb770009910 tx=0x7fb770009c20 comp rx=0 tx=0).stop 2024-04-27T04:03:24.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.369+0000 7fb7878e6700 1 -- 172.21.15.46:0/2739419016 shutdown_connections 2024-04-27T04:03:24.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.369+0000 7fb7878e6700 1 --2- 172.21.15.46:0/2739419016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb780104050 0x7fb780106440 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:24.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.369+0000 7fb7878e6700 1 --2- 172.21.15.46:0/2739419016 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb780101720 0x7fb780103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:24.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.369+0000 7fb7878e6700 1 -- 172.21.15.46:0/2739419016 >> 172.21.15.46:0/2739419016 conn(0x7fb7800fb110 msgr2=0x7fb7800fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:03:24.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.369+0000 7fb7878e6700 1 -- 172.21.15.46:0/2739419016 shutdown_connections 2024-04-27T04:03:24.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.369+0000 7fb7878e6700 1 -- 172.21.15.46:0/2739419016 wait complete. 2024-04-27T04:03:24.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.370+0000 7fb7878e6700 1 Processor -- start 2024-04-27T04:03:24.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.370+0000 7fb7878e6700 1 -- start start 2024-04-27T04:03:24.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.370+0000 7fb7878e6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb780101720 0x7fb780196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:24.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.370+0000 7fb7878e6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb780104050 0x7fb780196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:24.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.370+0000 7fb7878e6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb780197160 con 0x7fb780101720 2024-04-27T04:03:24.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.370+0000 7fb7878e6700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7801972a0 con 0x7fb780104050 2024-04-27T04:03:24.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.370+0000 7fb785682700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb780101720 0x7fb780196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:24.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.370+0000 7fb785682700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb780101720 0x7fb780196600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59162/0 (socket says 172.21.15.46:59162) 2024-04-27T04:03:24.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.370+0000 7fb785682700 1 -- 172.21.15.46:0/3555862832 learned_addr learned my addr 172.21.15.46:0/3555862832 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:03:24.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.370+0000 7fb784e81700 1 --2- 172.21.15.46:0/3555862832 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb780104050 0x7fb780196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:24.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.371+0000 7fb784e81700 1 -- 172.21.15.46:0/3555862832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb780101720 msgr2=0x7fb780196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.371+0000 7fb784e81700 1 --2- 172.21.15.46:0/3555862832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb780101720 0x7fb780196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.371+0000 7fb784e81700 1 -- 172.21.15.46:0/3555862832 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7700095f0 con 0x7fb780104050 2024-04-27T04:03:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.371+0000 7fb785682700 1 --2- 172.21.15.46:0/3555862832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb780101720 0x7fb780196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T04:03:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.371+0000 7fb784e81700 1 --2- 172.21.15.46:0/3555862832 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb780104050 0x7fb780196b40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb770006010 tx=0x7fb770005f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:03:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.371+0000 7fb7767fc700 1 -- 172.21.15.46:0/3555862832 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb77001d070 con 0x7fb780104050 2024-04-27T04:03:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.371+0000 7fb7767fc700 1 -- 172.21.15.46:0/3555862832 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7700052a0 con 0x7fb780104050 2024-04-27T04:03:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.371+0000 7fb7878e6700 1 -- 172.21.15.46:0/3555862832 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78019bca0 con 0x7fb780104050 2024-04-27T04:03:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.371+0000 7fb7767fc700 1 -- 172.21.15.46:0/3555862832 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb770017400 con 0x7fb780104050 2024-04-27T04:03:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.371+0000 7fb7878e6700 1 -- 172.21.15.46:0/3555862832 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78019c190 con 0x7fb780104050 2024-04-27T04:03:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.373+0000 7fb7878e6700 1 -- 172.21.15.46:0/3555862832 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78004ebb0 con 0x7fb780104050 2024-04-27T04:03:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.373+0000 7fb7767fc700 1 -- 172.21.15.46:0/3555862832 <== mon.1 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb770003780 con 0x7fb780104050 2024-04-27T04:03:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.373+0000 7fb7767fc700 1 --2- 172.21.15.46:0/3555862832 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb76c06cce0 0x7fb76c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.373+0000 7fb7767fc700 1 -- 172.21.15.46:0/3555862832 <== mon.1 v2:172.21.15.73:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb77008d620 con 0x7fb780104050 2024-04-27T04:03:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.373+0000 7fb785682700 1 --2- 172.21.15.46:0/3555862832 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb76c06cce0 0x7fb76c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.374+0000 7fb785682700 1 --2- 172.21.15.46:0/3555862832 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb76c06cce0 0x7fb76c06f1a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb77c011550 tx=0x7fb77c009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:03:24.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.378+0000 7fb7767fc700 1 -- 172.21.15.46:0/3555862832 <== mon.1 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fb77005b380 con 0x7fb780104050 2024-04-27T04:03:24.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.643+0000 7fb7878e6700 1 -- 172.21.15.46:0/3555862832 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb780105ee0 con 0x7fb76c06cce0 2024-04-27T04:03:24.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.644+0000 7fb7767fc700 1 -- 172.21.15.46:0/3555862832 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+349 (secure 0 0 0) 0x7fb780105ee0 con 0x7fb76c06cce0 2024-04-27T04:03:24.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.647+0000 7fb7878e6700 1 -- 172.21.15.46:0/3555862832 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb76c06cce0 msgr2=0x7fb76c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:24.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.647+0000 7fb7878e6700 1 --2- 172.21.15.46:0/3555862832 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb76c06cce0 0x7fb76c06f1a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb77c011550 tx=0x7fb77c009260 comp rx=0 tx=0).stop 2024-04-27T04:03:24.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.647+0000 7fb7878e6700 1 -- 172.21.15.46:0/3555862832 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb780104050 msgr2=0x7fb780196b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:24.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.647+0000 7fb7878e6700 1 --2- 172.21.15.46:0/3555862832 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb780104050 0x7fb780196b40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb770006010 tx=0x7fb770005f00 comp rx=0 tx=0).stop 2024-04-27T04:03:24.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.647+0000 7fb7878e6700 1 -- 172.21.15.46:0/3555862832 shutdown_connections 2024-04-27T04:03:24.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.647+0000 7fb7878e6700 1 --2- 172.21.15.46:0/3555862832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb780101720 0x7fb780196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:24.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.647+0000 7fb7878e6700 1 --2- 172.21.15.46:0/3555862832 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7fb76c06cce0 0x7fb76c06f1a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:24.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.647+0000 7fb7878e6700 1 --2- 172.21.15.46:0/3555862832 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb780104050 0x7fb780196b40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:24.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.647+0000 7fb7878e6700 1 -- 172.21.15.46:0/3555862832 >> 172.21.15.46:0/3555862832 conn(0x7fb7800fb110 msgr2=0x7fb7800fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:03:24.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.647+0000 7fb7878e6700 1 -- 172.21.15.46:0/3555862832 shutdown_connections 2024-04-27T04:03:24.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.647+0000 7fb7878e6700 1 -- 172.21.15.46:0/3555862832 wait complete. 2024-04-27T04:03:24.661 INFO:teuthology.orchestra.run.smithi046.stdout:true 2024-04-27T04:03:24.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.763+0000 7f9bde387700 1 -- 172.21.15.46:0/583976855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81040b0 msgr2=0x7f9bd81064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:24.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.763+0000 7f9bde387700 1 --2- 172.21.15.46:0/583976855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81040b0 0x7f9bd81064a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc009910 tx=0x7f9bcc009c20 comp rx=0 tx=0).stop 2024-04-27T04:03:24.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.763+0000 7f9bde387700 1 -- 172.21.15.46:0/583976855 shutdown_connections 2024-04-27T04:03:24.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.763+0000 7f9bde387700 1 --2- 172.21.15.46:0/583976855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81040b0 0x7f9bd81064a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:24.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.763+0000 7f9bde387700 1 --2- 172.21.15.46:0/583976855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8101780 0x7f9bd8103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:24.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.763+0000 7f9bde387700 1 -- 172.21.15.46:0/583976855 >> 172.21.15.46:0/583976855 conn(0x7f9bd80fb110 msgr2=0x7f9bd80fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:03:24.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.764+0000 7f9bde387700 1 -- 172.21.15.46:0/583976855 shutdown_connections 2024-04-27T04:03:24.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.764+0000 7f9bde387700 1 -- 172.21.15.46:0/583976855 wait complete. 2024-04-27T04:03:24.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.764+0000 7f9bde387700 1 Processor -- start 2024-04-27T04:03:24.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.764+0000 7f9bde387700 1 -- start start 2024-04-27T04:03:24.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.765+0000 7f9bde387700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8101780 0x7f9bd8101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:24.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.765+0000 7f9bde387700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81040b0 0x7f9bd80ff730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:24.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.765+0000 7f9bde387700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80ffc70 con 0x7f9bd81040b0 2024-04-27T04:03:24.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.765+0000 7f9bde387700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80ffdb0 con 0x7f9bd8101780 2024-04-27T04:03:24.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.765+0000 7f9bd77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81040b0 0x7f9bd80ff730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:24.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.765+0000 7f9bd77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81040b0 0x7f9bd80ff730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59190/0 (socket says 172.21.15.46:59190) 2024-04-27T04:03:24.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.765+0000 7f9bd77fe700 1 -- 172.21.15.46:0/2979590504 learned_addr learned my addr 172.21.15.46:0/2979590504 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:03:24.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.765+0000 7f9bd7fff700 1 --2- 172.21.15.46:0/2979590504 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8101780 0x7f9bd8101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:24.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.765+0000 7f9bd77fe700 1 -- 172.21.15.46:0/2979590504 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8101780 msgr2=0x7f9bd8101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:24.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.765+0000 7f9bd77fe700 1 --2- 172.21.15.46:0/2979590504 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8101780 0x7f9bd8101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:24.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.765+0000 7f9bd77fe700 1 -- 172.21.15.46:0/2979590504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc0095f0 con 0x7f9bd81040b0 2024-04-27T04:03:24.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.765+0000 7f9bd7fff700 1 --2- 172.21.15.46:0/2979590504 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8101780 0x7f9bd8101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T04:03:24.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.765+0000 7f9bd77fe700 1 --2- 172.21.15.46:0/2979590504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81040b0 0x7f9bd80ff730 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00b800 tx=0x7f9bcc005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:03:24.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.766+0000 7f9bd57fa700 1 -- 172.21.15.46:0/2979590504 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bcc01d070 con 0x7f9bd81040b0 2024-04-27T04:03:24.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.766+0000 7f9bde387700 1 -- 172.21.15.46:0/2979590504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd8100030 con 0x7f9bd81040b0 2024-04-27T04:03:24.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.766+0000 7f9bd57fa700 1 -- 172.21.15.46:0/2979590504 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bcc0052a0 con 0x7f9bd81040b0 2024-04-27T04:03:24.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.766+0000 7f9bd57fa700 1 -- 172.21.15.46:0/2979590504 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bcc017400 con 0x7f9bd81040b0 2024-04-27T04:03:24.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.766+0000 7f9bde387700 1 -- 172.21.15.46:0/2979590504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd8100520 con 0x7f9bd81040b0 2024-04-27T04:03:24.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.767+0000 7f9bde387700 1 -- 172.21.15.46:0/2979590504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd804ebb0 con 0x7f9bd81040b0 2024-04-27T04:03:24.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.767+0000 7f9bd57fa700 1 -- 172.21.15.46:0/2979590504 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9bcc003780 con 0x7f9bd81040b0 2024-04-27T04:03:24.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.768+0000 7f9bd57fa700 1 --2- 172.21.15.46:0/2979590504 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9bc006cce0 0x7f9bc006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:24.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.768+0000 7f9bd57fa700 1 -- 172.21.15.46:0/2979590504 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f9bcc08d600 con 0x7f9bd81040b0 2024-04-27T04:03:24.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.768+0000 7f9bd7fff700 1 --2- 172.21.15.46:0/2979590504 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9bc006cce0 0x7f9bc006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:24.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.768+0000 7f9bd7fff700 1 --2- 172.21.15.46:0/2979590504 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9bc006cce0 0x7f9bc006f1a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8011490 tx=0x7f9bc8009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:03:24.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:24.773+0000 7f9bd57fa700 1 -- 172.21.15.46:0/2979590504 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f9bcc05b480 con 0x7f9bd81040b0 2024-04-27T04:03:25.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.019+0000 7f9bde387700 1 -- 172.21.15.46:0/2979590504 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bd8105f40 con 0x7f9bc006cce0 2024-04-27T04:03:25.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.020+0000 7f9bd57fa700 1 -- 172.21.15.46:0/2979590504 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+349 (secure 0 0 0) 0x7f9bd8105f40 con 0x7f9bc006cce0 2024-04-27T04:03:25.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.023+0000 7f9bde387700 1 -- 172.21.15.46:0/2979590504 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9bc006cce0 msgr2=0x7f9bc006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:25.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.023+0000 7f9bde387700 1 --2- 172.21.15.46:0/2979590504 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9bc006cce0 0x7f9bc006f1a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8011490 tx=0x7f9bc8009260 comp rx=0 tx=0).stop 2024-04-27T04:03:25.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.023+0000 7f9bde387700 1 -- 172.21.15.46:0/2979590504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81040b0 msgr2=0x7f9bd80ff730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:25.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.023+0000 7f9bde387700 1 --2- 172.21.15.46:0/2979590504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81040b0 0x7f9bd80ff730 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00b800 tx=0x7f9bcc005f00 comp rx=0 tx=0).stop 2024-04-27T04:03:25.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.023+0000 7f9bde387700 1 -- 172.21.15.46:0/2979590504 shutdown_connections 2024-04-27T04:03:25.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.023+0000 7f9bde387700 1 --2- 172.21.15.46:0/2979590504 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f9bc006cce0 0x7f9bc006f1a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:25.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.023+0000 7f9bde387700 1 --2- 172.21.15.46:0/2979590504 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8101780 0x7f9bd8101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:25.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.023+0000 7f9bde387700 1 --2- 172.21.15.46:0/2979590504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81040b0 0x7f9bd80ff730 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:25.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.023+0000 7f9bde387700 1 -- 172.21.15.46:0/2979590504 >> 172.21.15.46:0/2979590504 conn(0x7f9bd80fb110 msgr2=0x7f9bd80fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:03:25.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.023+0000 7f9bde387700 1 -- 172.21.15.46:0/2979590504 shutdown_connections 2024-04-27T04:03:25.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.023+0000 7f9bde387700 1 -- 172.21.15.46:0/2979590504 wait complete. 2024-04-27T04:03:25.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.152+0000 7ff3053d6700 1 -- 172.21.15.46:0/2993533219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3000ffb10 msgr2=0x7ff3000fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:25.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.152+0000 7ff3053d6700 1 --2- 172.21.15.46:0/2993533219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3000ffb10 0x7ff3000fff30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009910 tx=0x7ff2f0009c20 comp rx=0 tx=0).stop 2024-04-27T04:03:25.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.152+0000 7ff3053d6700 1 -- 172.21.15.46:0/2993533219 shutdown_connections 2024-04-27T04:03:25.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.152+0000 7ff3053d6700 1 --2- 172.21.15.46:0/2993533219 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff300100d10 0x7ff300101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:25.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.152+0000 7ff3053d6700 1 --2- 172.21.15.46:0/2993533219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3000ffb10 0x7ff3000fff30 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:25.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.152+0000 7ff3053d6700 1 -- 172.21.15.46:0/2993533219 >> 172.21.15.46:0/2993533219 conn(0x7ff3000fb0b0 msgr2=0x7ff3000fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:03:25.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.152+0000 7ff3053d6700 1 -- 172.21.15.46:0/2993533219 shutdown_connections 2024-04-27T04:03:25.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.152+0000 7ff3053d6700 1 -- 172.21.15.46:0/2993533219 wait complete. 2024-04-27T04:03:25.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.153+0000 7ff3053d6700 1 Processor -- start 2024-04-27T04:03:25.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.153+0000 7ff3053d6700 1 -- start start 2024-04-27T04:03:25.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.153+0000 7ff3053d6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3000ffb10 0x7ff300069960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:25.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.153+0000 7ff3053d6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff300100d10 0x7ff30019d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:25.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.153+0000 7ff3053d6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff300197170 con 0x7ff3000ffb10 2024-04-27T04:03:25.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.153+0000 7ff3053d6700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30019dcd0 con 0x7ff300100d10 2024-04-27T04:03:25.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.153+0000 7ff2fe7fc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff300100d10 0x7ff30019d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:25.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.153+0000 7ff2fe7fc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff300100d10 0x7ff30019d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.46:39292/0 (socket says 172.21.15.46:39292) 2024-04-27T04:03:25.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.153+0000 7ff2fe7fc700 1 -- 172.21.15.46:0/3052881393 learned_addr learned my addr 172.21.15.46:0/3052881393 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:03:25.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.153+0000 7ff2feffd700 1 --2- 172.21.15.46:0/3052881393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3000ffb10 0x7ff300069960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:25.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.154+0000 7ff2fe7fc700 1 -- 172.21.15.46:0/3052881393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3000ffb10 msgr2=0x7ff300069960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:25.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.154+0000 7ff2fe7fc700 1 --2- 172.21.15.46:0/3052881393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3000ffb10 0x7ff300069960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:25.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.154+0000 7ff2fe7fc700 1 -- 172.21.15.46:0/3052881393 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f00095f0 con 0x7ff300100d10 2024-04-27T04:03:25.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.154+0000 7ff2fe7fc700 1 --2- 172.21.15.46:0/3052881393 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff300100d10 0x7ff30019d790 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff2f400e920 tx=0x7ff2f400ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:03:25.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.154+0000 7ff2e7fff700 1 -- 172.21.15.46:0/3052881393 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2f4009790 con 0x7ff300100d10 2024-04-27T04:03:25.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.154+0000 7ff3053d6700 1 -- 172.21.15.46:0/3052881393 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30019df30 con 0x7ff300100d10 2024-04-27T04:03:25.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.154+0000 7ff2e7fff700 1 -- 172.21.15.46:0/3052881393 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2f400cba0 con 0x7ff300100d10 2024-04-27T04:03:25.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.154+0000 7ff2e7fff700 1 -- 172.21.15.46:0/3052881393 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2f40186e0 con 0x7ff300100d10 2024-04-27T04:03:25.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.154+0000 7ff3053d6700 1 -- 172.21.15.46:0/3052881393 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30019e4d0 con 0x7ff300100d10 2024-04-27T04:03:25.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.156+0000 7ff3053d6700 1 -- 172.21.15.46:0/3052881393 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff30018f690 con 0x7ff300100d10 2024-04-27T04:03:25.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.156+0000 7ff2e7fff700 1 -- 172.21.15.46:0/3052881393 <== mon.1 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff2f4018840 con 0x7ff300100d10 2024-04-27T04:03:25.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.157+0000 7ff2e7fff700 1 --2- 172.21.15.46:0/3052881393 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff2e806ccf0 0x7ff2e806f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:25.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.157+0000 7ff2e7fff700 1 -- 172.21.15.46:0/3052881393 <== mon.1 v2:172.21.15.73:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff2f4014070 con 0x7ff300100d10 2024-04-27T04:03:25.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.157+0000 7ff2feffd700 1 --2- 172.21.15.46:0/3052881393 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff2e806ccf0 0x7ff2e806f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:25.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.158+0000 7ff2feffd700 1 --2- 172.21.15.46:0/3052881393 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff2e806ccf0 0x7ff2e806f1b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0006010 tx=0x7ff2f0009f70 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:03:25.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.161+0000 7ff2e7fff700 1 -- 172.21.15.46:0/3052881393 <== mon.1 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7ff2f4052c50 con 0x7ff300100d10 2024-04-27T04:03:25.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:24 smithi073 ceph-mon[34160]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:03:25.422 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:24 smithi046 ceph-mon[26899]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:03:25.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.529+0000 7ff3053d6700 1 -- 172.21.15.46:0/3052881393 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff30002cdb0 con 0x7ff2e806ccf0 2024-04-27T04:03:25.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.550+0000 7ff2e7fff700 1 -- 172.21.15.46:0/3052881393 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3864 (secure 0 0 0) 0x7ff30002cdb0 con 0x7ff2e806ccf0 2024-04-27T04:03:25.551 INFO:teuthology.orchestra.run.smithi046.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T04:03:25.551 INFO:teuthology.orchestra.run.smithi046.stdout:alertmanager.smithi046 smithi046 *:9093,9094 running (6m) 94s ago 7m 24.8M - 0.25.0 c8568f914cd2 698cffee765e 2024-04-27T04:03:25.551 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-exporter.smithi046 smithi046 running (7m) 94s ago 7m 7570k - 18.2.2-1260-g5403a021 267a473d505d 6eec9c927d3f 2024-04-27T04:03:25.551 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-exporter.smithi073 smithi073 running (6m) 98s ago 6m 7478k - 18.2.2-1260-g5403a021 267a473d505d d101eef4888d 2024-04-27T04:03:25.551 INFO:teuthology.orchestra.run.smithi046.stdout:crash.smithi046 smithi046 running (7m) 94s ago 7m 6660k - 18.2.2-1260-g5403a021 267a473d505d 338c449faabd 2024-04-27T04:03:25.551 INFO:teuthology.orchestra.run.smithi046.stdout:crash.smithi073 smithi073 running (6m) 98s ago 6m 6660k - 18.2.2-1260-g5403a021 267a473d505d be1aaee1c655 2024-04-27T04:03:25.552 INFO:teuthology.orchestra.run.smithi046.stdout:grafana.smithi046 smithi046 *:3000 running (5m) 94s ago 7m 84.6M - 9.4.7 954c08fa6188 a834a3988304 2024-04-27T04:03:25.552 INFO:teuthology.orchestra.run.smithi046.stdout:mds.cephfs.smithi046.mycakd smithi046 running (112s) 94s ago 112s 16.4M - 18.2.2-1260-g5403a021 267a473d505d 74d9a0b351ac 2024-04-27T04:03:25.552 INFO:teuthology.orchestra.run.smithi046.stdout:mds.cephfs.smithi046.qgmaql smithi046 running (2m) 94s ago 119s 12.9M - 18.2.2-1260-g5403a021 267a473d505d 741df55feeaf 2024-04-27T04:03:25.552 INFO:teuthology.orchestra.run.smithi046.stdout:mds.cephfs.smithi073.yryono smithi073 running (109s) 98s ago 109s 16.5M - 18.2.2-1260-g5403a021 267a473d505d 40d61213120b 2024-04-27T04:03:25.552 INFO:teuthology.orchestra.run.smithi046.stdout:mds.cephfs.smithi073.zkoypj smithi073 running (117s) 98s ago 116s 18.5M - 18.2.2-1260-g5403a021 267a473d505d 5812a431fb59 2024-04-27T04:03:25.552 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.smithi046.qrfdvr smithi046 *:9283,8765,8443 running (9m) 94s ago 9m 498M - 18.2.2-1260-g5403a021 267a473d505d ba1d99ba4a26 2024-04-27T04:03:25.552 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.smithi073.awyllt smithi073 *:8443,9283,8765 running (6m) 98s ago 6m 451M - 18.2.2-1260-g5403a021 267a473d505d 06769682a51f 2024-04-27T04:03:25.552 INFO:teuthology.orchestra.run.smithi046.stdout:mon.smithi046 smithi046 running (9m) 94s ago 9m 44.1M 2048M 18.2.2-1260-g5403a021 267a473d505d 72fac097effc 2024-04-27T04:03:25.552 INFO:teuthology.orchestra.run.smithi046.stdout:mon.smithi073 smithi073 running (6m) 98s ago 6m 35.2M 2048M 18.2.2-1260-g5403a021 267a473d505d 77769d199765 2024-04-27T04:03:25.552 INFO:teuthology.orchestra.run.smithi046.stdout:node-exporter.smithi046 smithi046 *:9100 running (7m) 94s ago 7m 16.3M - 1.5.0 0da6a335fe13 02c01746eff1 2024-04-27T04:03:25.552 INFO:teuthology.orchestra.run.smithi046.stdout:node-exporter.smithi073 smithi073 *:9100 running (6m) 98s ago 6m 15.2M - 1.5.0 0da6a335fe13 377ef4267056 2024-04-27T04:03:25.552 INFO:teuthology.orchestra.run.smithi046.stdout:osd.0 smithi046 running (5m) 94s ago 5m 57.4M 3948M 18.2.2-1260-g5403a021 267a473d505d 563c995c0df6 2024-04-27T04:03:25.552 INFO:teuthology.orchestra.run.smithi046.stdout:osd.1 smithi046 running (5m) 94s ago 5m 48.7M 3948M 18.2.2-1260-g5403a021 267a473d505d 5673c75e53bb 2024-04-27T04:03:25.553 INFO:teuthology.orchestra.run.smithi046.stdout:osd.2 smithi046 running (4m) 94s ago 4m 46.7M 3948M 18.2.2-1260-g5403a021 267a473d505d 9787b5012842 2024-04-27T04:03:25.553 INFO:teuthology.orchestra.run.smithi046.stdout:osd.3 smithi073 running (4m) 98s ago 4m 46.3M 4970M 18.2.2-1260-g5403a021 267a473d505d 0c89b358284d 2024-04-27T04:03:25.553 INFO:teuthology.orchestra.run.smithi046.stdout:osd.4 smithi073 running (3m) 98s ago 3m 53.4M 4970M 18.2.2-1260-g5403a021 267a473d505d 26f06c87c31e 2024-04-27T04:03:25.553 INFO:teuthology.orchestra.run.smithi046.stdout:osd.5 smithi073 running (3m) 98s ago 3m 49.6M 4970M 18.2.2-1260-g5403a021 267a473d505d e1ce06e0da9b 2024-04-27T04:03:25.553 INFO:teuthology.orchestra.run.smithi046.stdout:prometheus.smithi046 smithi046 *:9095 running (5m) 94s ago 7m 66.2M - 2.43.0 a07b618ecd1d e1a7d8656893 2024-04-27T04:03:25.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.552+0000 7ff3053d6700 1 -- 172.21.15.46:0/3052881393 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff2e806ccf0 msgr2=0x7ff2e806f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:25.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.552+0000 7ff3053d6700 1 --2- 172.21.15.46:0/3052881393 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff2e806ccf0 0x7ff2e806f1b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0006010 tx=0x7ff2f0009f70 comp rx=0 tx=0).stop 2024-04-27T04:03:25.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.552+0000 7ff3053d6700 1 -- 172.21.15.46:0/3052881393 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff300100d10 msgr2=0x7ff30019d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:25.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.552+0000 7ff3053d6700 1 --2- 172.21.15.46:0/3052881393 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff300100d10 0x7ff30019d790 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff2f400e920 tx=0x7ff2f400ece0 comp rx=0 tx=0).stop 2024-04-27T04:03:25.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.552+0000 7ff3053d6700 1 -- 172.21.15.46:0/3052881393 shutdown_connections 2024-04-27T04:03:25.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.552+0000 7ff3053d6700 1 --2- 172.21.15.46:0/3052881393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3000ffb10 0x7ff300069960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:25.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.552+0000 7ff3053d6700 1 --2- 172.21.15.46:0/3052881393 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7ff2e806ccf0 0x7ff2e806f1b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:25.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.552+0000 7ff3053d6700 1 --2- 172.21.15.46:0/3052881393 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff300100d10 0x7ff30019d790 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:25.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.552+0000 7ff3053d6700 1 -- 172.21.15.46:0/3052881393 >> 172.21.15.46:0/3052881393 conn(0x7ff3000fb0b0 msgr2=0x7ff3001042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:03:25.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.553+0000 7ff3053d6700 1 -- 172.21.15.46:0/3052881393 shutdown_connections 2024-04-27T04:03:25.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.553+0000 7ff3053d6700 1 -- 172.21.15.46:0/3052881393 wait complete. 2024-04-27T04:03:25.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.674+0000 7f16b42d1700 1 -- 172.21.15.46:0/3695782061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ac0ff4f0 msgr2=0x7f16ac0ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:25.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.674+0000 7f16b42d1700 1 --2- 172.21.15.46:0/3695782061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ac0ff4f0 0x7f16ac0ff910 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f16a8009910 tx=0x7f16a8009c20 comp rx=0 tx=0).stop 2024-04-27T04:03:25.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.674+0000 7f16b42d1700 1 -- 172.21.15.46:0/3695782061 shutdown_connections 2024-04-27T04:03:25.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.674+0000 7f16b42d1700 1 --2- 172.21.15.46:0/3695782061 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f16ac0ffe50 0x7f16ac1002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:25.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.674+0000 7f16b42d1700 1 --2- 172.21.15.46:0/3695782061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ac0ff4f0 0x7f16ac0ff910 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:25.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.674+0000 7f16b42d1700 1 -- 172.21.15.46:0/3695782061 >> 172.21.15.46:0/3695782061 conn(0x7f16ac0fb0b0 msgr2=0x7f16ac0fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:03:25.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.674+0000 7f16b42d1700 1 -- 172.21.15.46:0/3695782061 shutdown_connections 2024-04-27T04:03:25.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.674+0000 7f16b42d1700 1 -- 172.21.15.46:0/3695782061 wait complete. 2024-04-27T04:03:25.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.675+0000 7f16b42d1700 1 Processor -- start 2024-04-27T04:03:25.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.675+0000 7f16b42d1700 1 -- start start 2024-04-27T04:03:25.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.675+0000 7f16b42d1700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f16ac0ff4f0 0x7f16ac198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:25.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.675+0000 7f16b42d1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ac0ffe50 0x7f16ac198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:25.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.675+0000 7f16b42d1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac199390 con 0x7f16ac0ffe50 2024-04-27T04:03:25.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.675+0000 7f16b42d1700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac1994d0 con 0x7f16ac0ff4f0 2024-04-27T04:03:25.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.676+0000 7f16b186c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ac0ffe50 0x7f16ac198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:25.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.676+0000 7f16b206d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f16ac0ff4f0 0x7f16ac198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:25.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.676+0000 7f16b186c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ac0ffe50 0x7f16ac198d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59230/0 (socket says 172.21.15.46:59230) 2024-04-27T04:03:25.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.676+0000 7f16b186c700 1 -- 172.21.15.46:0/1878712745 learned_addr learned my addr 172.21.15.46:0/1878712745 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:03:25.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.676+0000 7f16b206d700 1 -- 172.21.15.46:0/1878712745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ac0ffe50 msgr2=0x7f16ac198d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:25.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.676+0000 7f16b206d700 1 --2- 172.21.15.46:0/1878712745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ac0ffe50 0x7f16ac198d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:25.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.676+0000 7f16b206d700 1 -- 172.21.15.46:0/1878712745 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a80095f0 con 0x7f16ac0ff4f0 2024-04-27T04:03:25.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.676+0000 7f16b186c700 1 --2- 172.21.15.46:0/1878712745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ac0ffe50 0x7f16ac198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T04:03:25.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.676+0000 7f16b206d700 1 --2- 172.21.15.46:0/1878712745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f16ac0ff4f0 0x7f16ac198830 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f16a8000c00 tx=0x7f16a800b600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:03:25.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.676+0000 7f16a37fe700 1 -- 172.21.15.46:0/1878712745 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16a801d070 con 0x7f16ac0ff4f0 2024-04-27T04:03:25.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.676+0000 7f16b42d1700 1 -- 172.21.15.46:0/1878712745 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ac19de80 con 0x7f16ac0ff4f0 2024-04-27T04:03:25.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.676+0000 7f16a37fe700 1 -- 172.21.15.46:0/1878712745 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16a80052a0 con 0x7f16ac0ff4f0 2024-04-27T04:03:25.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.677+0000 7f16a37fe700 1 -- 172.21.15.46:0/1878712745 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16a8017400 con 0x7f16ac0ff4f0 2024-04-27T04:03:25.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.677+0000 7f16b42d1700 1 -- 172.21.15.46:0/1878712745 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ac19e370 con 0x7f16ac0ff4f0 2024-04-27T04:03:25.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.678+0000 7f16b42d1700 1 -- 172.21.15.46:0/1878712745 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ac192990 con 0x7f16ac0ff4f0 2024-04-27T04:03:25.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.679+0000 7f16a37fe700 1 -- 172.21.15.46:0/1878712745 <== mon.1 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f16a800f7a0 con 0x7f16ac0ff4f0 2024-04-27T04:03:25.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.679+0000 7f16a37fe700 1 --2- 172.21.15.46:0/1878712745 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f169806cae0 0x7f169806efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:25.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.679+0000 7f16a37fe700 1 -- 172.21.15.46:0/1878712745 <== mon.1 v2:172.21.15.73:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f16a808cf20 con 0x7f16ac0ff4f0 2024-04-27T04:03:25.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.679+0000 7f16b186c700 1 --2- 172.21.15.46:0/1878712745 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f169806cae0 0x7f169806efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:25.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.680+0000 7f16b186c700 1 --2- 172.21.15.46:0/1878712745 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f169806cae0 0x7f169806efa0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f169c011550 tx=0x7f169c009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:03:25.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:25.684+0000 7f16a37fe700 1 -- 172.21.15.46:0/1878712745 <== mon.1 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f16a80c0c50 con 0x7f16ac0ff4f0 2024-04-27T04:03:26.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.005+0000 7f16b42d1700 1 -- 172.21.15.46:0/1878712745 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f16ac02cd40 con 0x7f16ac0ff4f0 2024-04-27T04:03:26.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.005+0000 7f16a37fe700 1 -- 172.21.15.46:0/1878712745 <== mon.1 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f16a805ad20 con 0x7f16ac0ff4f0 2024-04-27T04:03:26.007 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-27T04:03:26.007 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": { 2024-04-27T04:03:26.007 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)": 2 2024-04-27T04:03:26.007 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-27T04:03:26.007 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": { 2024-04-27T04:03:26.007 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)": 2 2024-04-27T04:03:26.007 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-27T04:03:26.007 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": { 2024-04-27T04:03:26.007 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)": 6 2024-04-27T04:03:26.007 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-27T04:03:26.007 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": { 2024-04-27T04:03:26.007 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)": 4 2024-04-27T04:03:26.008 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-27T04:03:26.008 INFO:teuthology.orchestra.run.smithi046.stdout: "overall": { 2024-04-27T04:03:26.008 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph version 18.2.2-1260-g5403a021 (5403a02127f27ec6c3309ef057522ab70cdeb9d2) reef (stable)": 14 2024-04-27T04:03:26.008 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-27T04:03:26.008 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-27T04:03:26.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.009+0000 7f16b42d1700 1 -- 172.21.15.46:0/1878712745 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f169806cae0 msgr2=0x7f169806efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:26.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.009+0000 7f16b42d1700 1 --2- 172.21.15.46:0/1878712745 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f169806cae0 0x7f169806efa0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f169c011550 tx=0x7f169c009260 comp rx=0 tx=0).stop 2024-04-27T04:03:26.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.009+0000 7f16b42d1700 1 -- 172.21.15.46:0/1878712745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f16ac0ff4f0 msgr2=0x7f16ac198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:26.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.009+0000 7f16b42d1700 1 --2- 172.21.15.46:0/1878712745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f16ac0ff4f0 0x7f16ac198830 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f16a8000c00 tx=0x7f16a800b600 comp rx=0 tx=0).stop 2024-04-27T04:03:26.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.009+0000 7f16b42d1700 1 -- 172.21.15.46:0/1878712745 shutdown_connections 2024-04-27T04:03:26.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.009+0000 7f16b42d1700 1 --2- 172.21.15.46:0/1878712745 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f169806cae0 0x7f169806efa0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:26.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.009+0000 7f16b42d1700 1 --2- 172.21.15.46:0/1878712745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f16ac0ff4f0 0x7f16ac198830 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:26.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.009+0000 7f16b42d1700 1 --2- 172.21.15.46:0/1878712745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ac0ffe50 0x7f16ac198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:26.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.009+0000 7f16b42d1700 1 -- 172.21.15.46:0/1878712745 >> 172.21.15.46:0/1878712745 conn(0x7f16ac0fb0b0 msgr2=0x7f16ac107f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:03:26.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.009+0000 7f16b42d1700 1 -- 172.21.15.46:0/1878712745 shutdown_connections 2024-04-27T04:03:26.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.009+0000 7f16b42d1700 1 -- 172.21.15.46:0/1878712745 wait complete. 2024-04-27T04:03:26.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:25 smithi073 ceph-mon[34160]: pgmap v241: 65 pgs: 65 active+clean; 26 MiB data, 238 MiB used, 536 GiB / 536 GiB avail; 684 KiB/s rd, 2.2 MiB/s wr, 110 op/s 2024-04-27T04:03:26.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:25 smithi073 ceph-mon[34160]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:03:26.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:25 smithi073 ceph-mon[34160]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:03:26.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.139+0000 7f135359e700 1 -- 172.21.15.46:0/3044715952 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1354101560 msgr2=0x7f1354103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:26.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.139+0000 7f135359e700 1 --2- 172.21.15.46:0/3044715952 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1354101560 0x7f1354103950 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1344009910 tx=0x7f1344009c20 comp rx=0 tx=0).stop 2024-04-27T04:03:26.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.139+0000 7f135359e700 1 -- 172.21.15.46:0/3044715952 shutdown_connections 2024-04-27T04:03:26.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.139+0000 7f135359e700 1 --2- 172.21.15.46:0/3044715952 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1354103e90 0x7f1354106280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:26.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.139+0000 7f135359e700 1 --2- 172.21.15.46:0/3044715952 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1354101560 0x7f1354103950 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:26.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.139+0000 7f135359e700 1 -- 172.21.15.46:0/3044715952 >> 172.21.15.46:0/3044715952 conn(0x7f13540faef0 msgr2=0x7f13540fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:03:26.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.139+0000 7f135359e700 1 -- 172.21.15.46:0/3044715952 shutdown_connections 2024-04-27T04:03:26.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.139+0000 7f135359e700 1 -- 172.21.15.46:0/3044715952 wait complete. 2024-04-27T04:03:26.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.140+0000 7f135359e700 1 Processor -- start 2024-04-27T04:03:26.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.140+0000 7f135359e700 1 -- start start 2024-04-27T04:03:26.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.140+0000 7f135359e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1354101560 0x7f13541963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:26.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.140+0000 7f135359e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1354103e90 0x7f1354196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:26.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.140+0000 7f135359e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1354196f40 con 0x7f1354101560 2024-04-27T04:03:26.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.140+0000 7f135359e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1354197080 con 0x7f1354103e90 2024-04-27T04:03:26.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.140+0000 7f135259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1354101560 0x7f13541963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:26.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.140+0000 7f135259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1354101560 0x7f13541963e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59246/0 (socket says 172.21.15.46:59246) 2024-04-27T04:03:26.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.140+0000 7f135259c700 1 -- 172.21.15.46:0/1795411139 learned_addr learned my addr 172.21.15.46:0/1795411139 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:03:26.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.140+0000 7f1351d9b700 1 --2- 172.21.15.46:0/1795411139 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1354103e90 0x7f1354196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:26.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.141+0000 7f1351d9b700 1 -- 172.21.15.46:0/1795411139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1354101560 msgr2=0x7f13541963e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:26.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.141+0000 7f1351d9b700 1 --2- 172.21.15.46:0/1795411139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1354101560 0x7f13541963e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:26.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.141+0000 7f1351d9b700 1 -- 172.21.15.46:0/1795411139 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13440095f0 con 0x7f1354103e90 2024-04-27T04:03:26.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.141+0000 7f135259c700 1 --2- 172.21.15.46:0/1795411139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1354101560 0x7f13541963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T04:03:26.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.141+0000 7f1351d9b700 1 --2- 172.21.15.46:0/1795411139 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1354103e90 0x7f1354196920 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f134800e970 tx=0x7f134800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:03:26.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.141+0000 7f13437fe700 1 -- 172.21.15.46:0/1795411139 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f134800c9d0 con 0x7f1354103e90 2024-04-27T04:03:26.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.141+0000 7f13437fe700 1 -- 172.21.15.46:0/1795411139 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1348010430 con 0x7f1354103e90 2024-04-27T04:03:26.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.141+0000 7f13437fe700 1 -- 172.21.15.46:0/1795411139 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1348018630 con 0x7f1354103e90 2024-04-27T04:03:26.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.141+0000 7f135359e700 1 -- 172.21.15.46:0/1795411139 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f135419bae0 con 0x7f1354103e90 2024-04-27T04:03:26.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.141+0000 7f135359e700 1 -- 172.21.15.46:0/1795411139 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135419c0b0 con 0x7f1354103e90 2024-04-27T04:03:26.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.142+0000 7f135359e700 1 -- 172.21.15.46:0/1795411139 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135404f400 con 0x7f1354103e90 2024-04-27T04:03:26.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.143+0000 7f13437fe700 1 -- 172.21.15.46:0/1795411139 <== mon.1 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f13480105a0 con 0x7f1354103e90 2024-04-27T04:03:26.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.143+0000 7f13437fe700 1 --2- 172.21.15.46:0/1795411139 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f133c06cce0 0x7f133c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:26.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.143+0000 7f13437fe700 1 -- 172.21.15.46:0/1795411139 <== mon.1 v2:172.21.15.73:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1348014070 con 0x7f1354103e90 2024-04-27T04:03:26.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.143+0000 7f135259c700 1 --2- 172.21.15.46:0/1795411139 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f133c06cce0 0x7f133c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:26.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.144+0000 7f135259c700 1 --2- 172.21.15.46:0/1795411139 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f133c06cce0 0x7f133c06f1a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1344009fd0 tx=0x7f1344005940 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:03:26.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.148+0000 7f13437fe700 1 -- 172.21.15.46:0/1795411139 <== mon.1 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f1348052c50 con 0x7f1354103e90 2024-04-27T04:03:26.422 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:25 smithi046 ceph-mon[26899]: pgmap v241: 65 pgs: 65 active+clean; 26 MiB data, 238 MiB used, 536 GiB / 536 GiB avail; 684 KiB/s rd, 2.2 MiB/s wr, 110 op/s 2024-04-27T04:03:26.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:25 smithi046 ceph-mon[26899]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:03:26.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:25 smithi046 ceph-mon[26899]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:03:26.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.507+0000 7f135359e700 1 -- 172.21.15.46:0/1795411139 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f13541095d0 con 0x7f1354103e90 2024-04-27T04:03:26.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.508+0000 7f13437fe700 1 -- 172.21.15.46:0/1795411139 <== mon.1 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1865 (secure 0 0 0) 0x7f134805a270 con 0x7f1354103e90 2024-04-27T04:03:26.508 INFO:teuthology.orchestra.run.smithi046.stdout:e19 2024-04-27T04:03:26.508 INFO:teuthology.orchestra.run.smithi046.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T04:03:26.508 INFO:teuthology.orchestra.run.smithi046.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T04:03:26.508 INFO:teuthology.orchestra.run.smithi046.stdout:legacy client fscid: 1 2024-04-27T04:03:26.508 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:03:26.509 INFO:teuthology.orchestra.run.smithi046.stdout:Filesystem 'cephfs' (1) 2024-04-27T04:03:26.509 INFO:teuthology.orchestra.run.smithi046.stdout:fs_name cephfs 2024-04-27T04:03:26.509 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 19 2024-04-27T04:03:26.509 INFO:teuthology.orchestra.run.smithi046.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-27T04:03:26.509 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-04-27T04:01:21.676811+0000 2024-04-27T04:03:26.509 INFO:teuthology.orchestra.run.smithi046.stdout:modified 2024-04-27T04:01:42.132288+0000 2024-04-27T04:03:26.509 INFO:teuthology.orchestra.run.smithi046.stdout:tableserver 0 2024-04-27T04:03:26.509 INFO:teuthology.orchestra.run.smithi046.stdout:root 0 2024-04-27T04:03:26.509 INFO:teuthology.orchestra.run.smithi046.stdout:session_timeout 60 2024-04-27T04:03:26.509 INFO:teuthology.orchestra.run.smithi046.stdout:session_autoclose 300 2024-04-27T04:03:26.509 INFO:teuthology.orchestra.run.smithi046.stdout:max_file_size 1099511627776 2024-04-27T04:03:26.509 INFO:teuthology.orchestra.run.smithi046.stdout:max_xattr_size 65536 2024-04-27T04:03:26.509 INFO:teuthology.orchestra.run.smithi046.stdout:required_client_features {} 2024-04-27T04:03:26.510 INFO:teuthology.orchestra.run.smithi046.stdout:last_failure 0 2024-04-27T04:03:26.510 INFO:teuthology.orchestra.run.smithi046.stdout:last_failure_osd_epoch 46 2024-04-27T04:03:26.510 INFO:teuthology.orchestra.run.smithi046.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T04:03:26.510 INFO:teuthology.orchestra.run.smithi046.stdout:max_mds 2 2024-04-27T04:03:26.510 INFO:teuthology.orchestra.run.smithi046.stdout:in 0,1 2024-04-27T04:03:26.510 INFO:teuthology.orchestra.run.smithi046.stdout:up {0=14488,1=24285} 2024-04-27T04:03:26.510 INFO:teuthology.orchestra.run.smithi046.stdout:failed 2024-04-27T04:03:26.510 INFO:teuthology.orchestra.run.smithi046.stdout:damaged 2024-04-27T04:03:26.510 INFO:teuthology.orchestra.run.smithi046.stdout:stopped 2024-04-27T04:03:26.510 INFO:teuthology.orchestra.run.smithi046.stdout:data_pools [3] 2024-04-27T04:03:26.510 INFO:teuthology.orchestra.run.smithi046.stdout:metadata_pool 2 2024-04-27T04:03:26.510 INFO:teuthology.orchestra.run.smithi046.stdout:inline_data enabled 2024-04-27T04:03:26.510 INFO:teuthology.orchestra.run.smithi046.stdout:balancer 2024-04-27T04:03:26.510 INFO:teuthology.orchestra.run.smithi046.stdout:bal_rank_mask -1 2024-04-27T04:03:26.511 INFO:teuthology.orchestra.run.smithi046.stdout:standby_count_wanted 1 2024-04-27T04:03:26.511 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.cephfs.smithi046.mycakd{0:14488} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.46:6828/3081610305,v1:172.21.15.46:6829/3081610305] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T04:03:26.511 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.cephfs.smithi046.qgmaql{0:24303} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.46:6826/2653255940,v1:172.21.15.46:6827/2653255940] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T04:03:26.511 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.cephfs.smithi073.zkoypj{1:24285} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.73:6824/3397960510,v1:172.21.15.73:6825/3397960510] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T04:03:26.511 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.cephfs.smithi073.yryono{1:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.73:6826/565262827,v1:172.21.15.73:6827/565262827] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T04:03:26.511 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:03:26.511 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-27T04:03:26.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.510+0000 7f135359e700 1 -- 172.21.15.46:0/1795411139 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f133c06cce0 msgr2=0x7f133c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:26.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.510+0000 7f135359e700 1 --2- 172.21.15.46:0/1795411139 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f133c06cce0 0x7f133c06f1a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1344009fd0 tx=0x7f1344005940 comp rx=0 tx=0).stop 2024-04-27T04:03:26.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.510+0000 7f135359e700 1 -- 172.21.15.46:0/1795411139 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1354103e90 msgr2=0x7f1354196920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:26.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.510+0000 7f135359e700 1 --2- 172.21.15.46:0/1795411139 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1354103e90 0x7f1354196920 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f134800e970 tx=0x7f134800ec80 comp rx=0 tx=0).stop 2024-04-27T04:03:26.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.510+0000 7f135359e700 1 -- 172.21.15.46:0/1795411139 shutdown_connections 2024-04-27T04:03:26.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.510+0000 7f135359e700 1 --2- 172.21.15.46:0/1795411139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1354101560 0x7f13541963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:26.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.510+0000 7f135359e700 1 --2- 172.21.15.46:0/1795411139 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f133c06cce0 0x7f133c06f1a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:26.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.510+0000 7f135359e700 1 --2- 172.21.15.46:0/1795411139 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1354103e90 0x7f1354196920 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:26.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.510+0000 7f135359e700 1 -- 172.21.15.46:0/1795411139 >> 172.21.15.46:0/1795411139 conn(0x7f13540faef0 msgr2=0x7f13540fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:03:26.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.510+0000 7f135359e700 1 -- 172.21.15.46:0/1795411139 shutdown_connections 2024-04-27T04:03:26.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.510+0000 7f135359e700 1 -- 172.21.15.46:0/1795411139 wait complete. 2024-04-27T04:03:26.513 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 19 2024-04-27T04:03:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.635+0000 7f8315928700 1 -- 172.21.15.46:0/379817801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83100ffef0 msgr2=0x7f8310100370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.635+0000 7f8315928700 1 --2- 172.21.15.46:0/379817801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83100ffef0 0x7f8310100370 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8304009910 tx=0x7f8304009c20 comp rx=0 tx=0).stop 2024-04-27T04:03:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.635+0000 7f8315928700 1 -- 172.21.15.46:0/379817801 shutdown_connections 2024-04-27T04:03:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.635+0000 7f8315928700 1 --2- 172.21.15.46:0/379817801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83100ffef0 0x7f8310100370 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.635+0000 7f8315928700 1 --2- 172.21.15.46:0/379817801 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f83100ff590 0x7f83100ff9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.635+0000 7f8315928700 1 -- 172.21.15.46:0/379817801 >> 172.21.15.46:0/379817801 conn(0x7f83100fb110 msgr2=0x7f83100fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:03:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.635+0000 7f8315928700 1 -- 172.21.15.46:0/379817801 shutdown_connections 2024-04-27T04:03:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.635+0000 7f8315928700 1 -- 172.21.15.46:0/379817801 wait complete. 2024-04-27T04:03:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.636+0000 7f8315928700 1 Processor -- start 2024-04-27T04:03:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.636+0000 7f8315928700 1 -- start start 2024-04-27T04:03:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.636+0000 7f8315928700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f83100ff590 0x7f8310198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.636+0000 7f8315928700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83100ffef0 0x7f8310198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.636+0000 7f8315928700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83101993b0 con 0x7f83100ffef0 2024-04-27T04:03:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.636+0000 7f8315928700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83101994f0 con 0x7f83100ff590 2024-04-27T04:03:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.636+0000 7f830e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83100ffef0 0x7f8310198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.636+0000 7f830e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83100ffef0 0x7f8310198d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59260/0 (socket says 172.21.15.46:59260) 2024-04-27T04:03:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.636+0000 7f830e7fc700 1 -- 172.21.15.46:0/2837984482 learned_addr learned my addr 172.21.15.46:0/2837984482 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:03:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.637+0000 7f830effd700 1 --2- 172.21.15.46:0/2837984482 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f83100ff590 0x7f8310198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.637+0000 7f830e7fc700 1 -- 172.21.15.46:0/2837984482 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f83100ff590 msgr2=0x7f8310198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.637+0000 7f830e7fc700 1 --2- 172.21.15.46:0/2837984482 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f83100ff590 0x7f8310198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.637+0000 7f830e7fc700 1 -- 172.21.15.46:0/2837984482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83040095f0 con 0x7f83100ffef0 2024-04-27T04:03:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.637+0000 7f830e7fc700 1 --2- 172.21.15.46:0/2837984482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83100ffef0 0x7f8310198d90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8304005b40 tx=0x7f830400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:03:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.637+0000 7f8314926700 1 -- 172.21.15.46:0/2837984482 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f830401d070 con 0x7f83100ffef0 2024-04-27T04:03:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.637+0000 7f8315928700 1 -- 172.21.15.46:0/2837984482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f831019def0 con 0x7f83100ffef0 2024-04-27T04:03:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.637+0000 7f8314926700 1 -- 172.21.15.46:0/2837984482 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83040050d0 con 0x7f83100ffef0 2024-04-27T04:03:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.638+0000 7f8314926700 1 -- 172.21.15.46:0/2837984482 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8304022470 con 0x7f83100ffef0 2024-04-27T04:03:26.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.638+0000 7f8315928700 1 -- 172.21.15.46:0/2837984482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f831019e3b0 con 0x7f83100ffef0 2024-04-27T04:03:26.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.639+0000 7f8315928700 1 -- 172.21.15.46:0/2837984482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8310192a10 con 0x7f83100ffef0 2024-04-27T04:03:26.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.639+0000 7f8314926700 1 -- 172.21.15.46:0/2837984482 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8304017400 con 0x7f83100ffef0 2024-04-27T04:03:26.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.640+0000 7f8314926700 1 --2- 172.21.15.46:0/2837984482 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f82f806ccf0 0x7f82f806f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:26.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.640+0000 7f8314926700 1 -- 172.21.15.46:0/2837984482 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f830408d740 con 0x7f83100ffef0 2024-04-27T04:03:26.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.640+0000 7f830effd700 1 --2- 172.21.15.46:0/2837984482 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f82f806ccf0 0x7f82f806f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:26.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.641+0000 7f830effd700 1 --2- 172.21.15.46:0/2837984482 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f82f806ccf0 0x7f82f806f1b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8300009c60 tx=0x7f8300009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:03:26.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:26.648+0000 7f8314926700 1 -- 172.21.15.46:0/2837984482 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f830405b5c0 con 0x7f83100ffef0 2024-04-27T04:03:27.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.011+0000 7f8315928700 1 -- 172.21.15.46:0/2837984482 --> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f831002cf00 con 0x7f82f806ccf0 2024-04-27T04:03:27.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.012+0000 7f8314926700 1 -- 172.21.15.46:0/2837984482 <== mgr.14217 v2:172.21.15.46:6800/4118816988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+328 (secure 0 0 0) 0x7f831002cf00 con 0x7f82f806ccf0 2024-04-27T04:03:27.012 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-27T04:03:27.012 INFO:teuthology.orchestra.run.smithi046.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:90a3a9fa9ceb8977f6dad9ccd36fe46018d07e2118199820523a6b35301d9c16", 2024-04-27T04:03:27.013 INFO:teuthology.orchestra.run.smithi046.stdout: "in_progress": true, 2024-04-27T04:03:27.013 INFO:teuthology.orchestra.run.smithi046.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T04:03:27.013 INFO:teuthology.orchestra.run.smithi046.stdout: "services_complete": [], 2024-04-27T04:03:27.013 INFO:teuthology.orchestra.run.smithi046.stdout: "progress": "", 2024-04-27T04:03:27.013 INFO:teuthology.orchestra.run.smithi046.stdout: "message": "Currently upgrading mgr daemons", 2024-04-27T04:03:27.013 INFO:teuthology.orchestra.run.smithi046.stdout: "is_paused": false 2024-04-27T04:03:27.013 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-27T04:03:27.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.015+0000 7f8315928700 1 -- 172.21.15.46:0/2837984482 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f82f806ccf0 msgr2=0x7f82f806f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:27.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.015+0000 7f8315928700 1 --2- 172.21.15.46:0/2837984482 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f82f806ccf0 0x7f82f806f1b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8300009c60 tx=0x7f8300009260 comp rx=0 tx=0).stop 2024-04-27T04:03:27.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.015+0000 7f8315928700 1 -- 172.21.15.46:0/2837984482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83100ffef0 msgr2=0x7f8310198d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.015+0000 7f8315928700 1 --2- 172.21.15.46:0/2837984482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83100ffef0 0x7f8310198d90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8304005b40 tx=0x7f830400f690 comp rx=0 tx=0).stop 2024-04-27T04:03:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.015+0000 7f8315928700 1 -- 172.21.15.46:0/2837984482 shutdown_connections 2024-04-27T04:03:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.015+0000 7f8315928700 1 --2- 172.21.15.46:0/2837984482 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f82f806ccf0 0x7f82f806f1b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.015+0000 7f8315928700 1 --2- 172.21.15.46:0/2837984482 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f83100ff590 0x7f8310198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.015+0000 7f8315928700 1 --2- 172.21.15.46:0/2837984482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83100ffef0 0x7f8310198d90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.015+0000 7f8315928700 1 -- 172.21.15.46:0/2837984482 >> 172.21.15.46:0/2837984482 conn(0x7f83100fb110 msgr2=0x7f8310107fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:03:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.015+0000 7f8315928700 1 -- 172.21.15.46:0/2837984482 shutdown_connections 2024-04-27T04:03:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.015+0000 7f8315928700 1 -- 172.21.15.46:0/2837984482 wait complete. 2024-04-27T04:03:27.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.134+0000 7f7fbbf15700 1 -- 172.21.15.46:0/76624942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb4104050 msgr2=0x7f7fb4106440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:27.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.134+0000 7f7fbbf15700 1 --2- 172.21.15.46:0/76624942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb4104050 0x7f7fb4106440 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4009910 tx=0x7f7fa4009c20 comp rx=0 tx=0).stop 2024-04-27T04:03:27.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.134+0000 7f7fbbf15700 1 -- 172.21.15.46:0/76624942 shutdown_connections 2024-04-27T04:03:27.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.134+0000 7f7fbbf15700 1 --2- 172.21.15.46:0/76624942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb4104050 0x7f7fb4106440 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:27.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.134+0000 7f7fbbf15700 1 --2- 172.21.15.46:0/76624942 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fb4101720 0x7f7fb4103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:27.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.134+0000 7f7fbbf15700 1 -- 172.21.15.46:0/76624942 >> 172.21.15.46:0/76624942 conn(0x7f7fb40fb110 msgr2=0x7f7fb40fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:03:27.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.134+0000 7f7fbbf15700 1 -- 172.21.15.46:0/76624942 shutdown_connections 2024-04-27T04:03:27.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.134+0000 7f7fbbf15700 1 -- 172.21.15.46:0/76624942 wait complete. 2024-04-27T04:03:27.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.135+0000 7f7fbbf15700 1 Processor -- start 2024-04-27T04:03:27.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.135+0000 7f7fbbf15700 1 -- start start 2024-04-27T04:03:27.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.135+0000 7f7fbbf15700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb4101720 0x7f7fb4101030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:27.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.135+0000 7f7fbbf15700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fb4104050 0x7f7fb40ff6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:27.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.135+0000 7f7fbbf15700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb40ffc10 con 0x7f7fb4101720 2024-04-27T04:03:27.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.135+0000 7f7fbbf15700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb40ffd50 con 0x7f7fb4104050 2024-04-27T04:03:27.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.135+0000 7f7fb9cb1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb4101720 0x7f7fb4101030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:27.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.135+0000 7f7fb9cb1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb4101720 0x7f7fb4101030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59286/0 (socket says 172.21.15.46:59286) 2024-04-27T04:03:27.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.135+0000 7f7fb9cb1700 1 -- 172.21.15.46:0/2598582181 learned_addr learned my addr 172.21.15.46:0/2598582181 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:03:27.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.136+0000 7f7fb94b0700 1 --2- 172.21.15.46:0/2598582181 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fb4104050 0x7f7fb40ff6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:27.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.136+0000 7f7fb9cb1700 1 -- 172.21.15.46:0/2598582181 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fb4104050 msgr2=0x7f7fb40ff6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:27.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.136+0000 7f7fb9cb1700 1 --2- 172.21.15.46:0/2598582181 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fb4104050 0x7f7fb40ff6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:27.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.136+0000 7f7fb9cb1700 1 -- 172.21.15.46:0/2598582181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa40095f0 con 0x7f7fb4101720 2024-04-27T04:03:27.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.136+0000 7f7fb94b0700 1 --2- 172.21.15.46:0/2598582181 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fb4104050 0x7f7fb40ff6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T04:03:27.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.136+0000 7f7fb9cb1700 1 --2- 172.21.15.46:0/2598582181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb4101720 0x7f7fb4101030 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000e920 tx=0x7f7fb000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:03:27.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.136+0000 7f7faaffd700 1 -- 172.21.15.46:0/2598582181 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fb000c9d0 con 0x7f7fb4101720 2024-04-27T04:03:27.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.136+0000 7f7fbbf15700 1 -- 172.21.15.46:0/2598582181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb4100030 con 0x7f7fb4101720 2024-04-27T04:03:27.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.136+0000 7f7fbbf15700 1 -- 172.21.15.46:0/2598582181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb419e660 con 0x7f7fb4101720 2024-04-27T04:03:27.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.136+0000 7f7faaffd700 1 -- 172.21.15.46:0/2598582181 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7fb0010430 con 0x7f7fb4101720 2024-04-27T04:03:27.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.136+0000 7f7faaffd700 1 -- 172.21.15.46:0/2598582181 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fb00185f0 con 0x7f7fb4101720 2024-04-27T04:03:27.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.138+0000 7f7fbbf15700 1 -- 172.21.15.46:0/2598582181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f98005320 con 0x7f7fb4101720 2024-04-27T04:03:27.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.139+0000 7f7faaffd700 1 -- 172.21.15.46:0/2598582181 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7fb0010a80 con 0x7f7fb4101720 2024-04-27T04:03:27.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.139+0000 7f7faaffd700 1 --2- 172.21.15.46:0/2598582181 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7fa006ccf0 0x7f7fa006f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:27.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.139+0000 7f7faaffd700 1 -- 172.21.15.46:0/2598582181 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7fb0014070 con 0x7f7fb4101720 2024-04-27T04:03:27.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.139+0000 7f7fb94b0700 1 --2- 172.21.15.46:0/2598582181 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7fa006ccf0 0x7f7fa006f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:27.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.140+0000 7f7fb94b0700 1 --2- 172.21.15.46:0/2598582181 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7fa006ccf0 0x7f7fa006f1b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400b800 tx=0x7f7fa4009f70 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:03:27.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.144+0000 7f7faaffd700 1 -- 172.21.15.46:0/2598582181 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f7fb0052c50 con 0x7f7fb4101720 2024-04-27T04:03:27.274 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:26 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/1878712745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T04:03:27.275 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:26 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:27.275 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:26 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:27.275 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:26 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:03:27.275 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:26 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/1795411139' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T04:03:27.422 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:26 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/1878712745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T04:03:27.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:26 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:27.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:26 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:27.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:26 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:03:27.423 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:26 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/1795411139' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T04:03:27.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.602+0000 7f7fbbf15700 1 -- 172.21.15.46:0/2598582181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7f98005190 con 0x7f7fb4101720 2024-04-27T04:03:27.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.603+0000 7f7faaffd700 1 -- 172.21.15.46:0/2598582181 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f7fb005a2c0 con 0x7f7fb4101720 2024-04-27T04:03:27.603 INFO:teuthology.orchestra.run.smithi046.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-27T04:03:27.603 INFO:teuthology.orchestra.run.smithi046.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-27T04:03:27.603 INFO:teuthology.orchestra.run.smithi046.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-27T04:03:27.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.605+0000 7f7fbbf15700 1 -- 172.21.15.46:0/2598582181 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7fa006ccf0 msgr2=0x7f7fa006f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:27.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.605+0000 7f7fbbf15700 1 --2- 172.21.15.46:0/2598582181 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7fa006ccf0 0x7f7fa006f1b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400b800 tx=0x7f7fa4009f70 comp rx=0 tx=0).stop 2024-04-27T04:03:27.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.605+0000 7f7fbbf15700 1 -- 172.21.15.46:0/2598582181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb4101720 msgr2=0x7f7fb4101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:27.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.605+0000 7f7fbbf15700 1 --2- 172.21.15.46:0/2598582181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb4101720 0x7f7fb4101030 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000e920 tx=0x7f7fb000ec30 comp rx=0 tx=0).stop 2024-04-27T04:03:27.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.605+0000 7f7fbbf15700 1 -- 172.21.15.46:0/2598582181 shutdown_connections 2024-04-27T04:03:27.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.605+0000 7f7fbbf15700 1 --2- 172.21.15.46:0/2598582181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb4101720 0x7f7fb4101030 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:27.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.605+0000 7f7fbbf15700 1 --2- 172.21.15.46:0/2598582181 >> [v2:172.21.15.46:6800/4118816988,v1:172.21.15.46:6801/4118816988] conn(0x7f7fa006ccf0 0x7f7fa006f1b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:27.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.605+0000 7f7fbbf15700 1 --2- 172.21.15.46:0/2598582181 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fb4104050 0x7f7fb40ff6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:27.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.605+0000 7f7fbbf15700 1 -- 172.21.15.46:0/2598582181 >> 172.21.15.46:0/2598582181 conn(0x7f7fb40fb110 msgr2=0x7f7fb40fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:03:27.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.605+0000 7f7fbbf15700 1 -- 172.21.15.46:0/2598582181 shutdown_connections 2024-04-27T04:03:27.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:27.605+0000 7f7fbbf15700 1 -- 172.21.15.46:0/2598582181 wait complete. 2024-04-27T04:03:28.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:28 smithi073 ceph-mon[34160]: pgmap v242: 65 pgs: 65 active+clean; 54 MiB data, 377 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 404 op/s 2024-04-27T04:03:28.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:28 smithi073 ceph-mon[34160]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:03:28.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:28 smithi073 ceph-mon[34160]: from='client.? 172.21.15.46:0/2598582181' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T04:03:28.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:28 smithi046 ceph-mon[26899]: pgmap v242: 65 pgs: 65 active+clean; 54 MiB data, 377 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 404 op/s 2024-04-27T04:03:28.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:28 smithi046 ceph-mon[26899]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T04:03:28.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:28 smithi046 ceph-mon[26899]: from='client.? 172.21.15.46:0/2598582181' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T04:03:30.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:30 smithi073 ceph-mon[34160]: pgmap v243: 65 pgs: 65 active+clean; 62 MiB data, 431 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 5.4 MiB/s wr, 498 op/s 2024-04-27T04:03:30.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:30 smithi046 ceph-mon[26899]: pgmap v243: 65 pgs: 65 active+clean; 62 MiB data, 431 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 5.4 MiB/s wr, 498 op/s 2024-04-27T04:03:31.556 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:31 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:03:31.574 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:31 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:03:32.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:32 smithi073 ceph-mon[34160]: pgmap v244: 65 pgs: 65 active+clean; 66 MiB data, 447 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.8 MiB/s wr, 499 op/s 2024-04-27T04:03:32.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:32 smithi073 ceph-mon[34160]: Standby manager daemon smithi073.awyllt restarted 2024-04-27T04:03:32.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:32 smithi073 ceph-mon[34160]: Standby manager daemon smithi073.awyllt started 2024-04-27T04:03:32.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:32 smithi073 ceph-mon[34160]: from='mgr.? 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi073.awyllt/crt"}]: dispatch 2024-04-27T04:03:32.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:32 smithi073 ceph-mon[34160]: from='mgr.? 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T04:03:32.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:32 smithi073 ceph-mon[34160]: from='mgr.? 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi073.awyllt/key"}]: dispatch 2024-04-27T04:03:32.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:32 smithi073 ceph-mon[34160]: from='mgr.? 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T04:03:32.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:32 smithi046 ceph-mon[26899]: pgmap v244: 65 pgs: 65 active+clean; 66 MiB data, 447 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.8 MiB/s wr, 499 op/s 2024-04-27T04:03:32.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:32 smithi046 ceph-mon[26899]: Standby manager daemon smithi073.awyllt restarted 2024-04-27T04:03:32.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:32 smithi046 ceph-mon[26899]: Standby manager daemon smithi073.awyllt started 2024-04-27T04:03:32.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:32 smithi046 ceph-mon[26899]: from='mgr.? 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi073.awyllt/crt"}]: dispatch 2024-04-27T04:03:32.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:32 smithi046 ceph-mon[26899]: from='mgr.? 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T04:03:32.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:32 smithi046 ceph-mon[26899]: from='mgr.? 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi073.awyllt/key"}]: dispatch 2024-04-27T04:03:32.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:32 smithi046 ceph-mon[26899]: from='mgr.? 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T04:03:33.422 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:33 smithi046 ceph-mon[26899]: mgrmap e20: smithi046.qrfdvr(active, since 7m), standbys: smithi073.awyllt 2024-04-27T04:03:33.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:33 smithi073 ceph-mon[34160]: mgrmap e20: smithi046.qrfdvr(active, since 7m), standbys: smithi073.awyllt 2024-04-27T04:03:34.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:34 smithi073 ceph-mon[34160]: pgmap v245: 65 pgs: 65 active+clean; 77 MiB data, 456 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 6.7 MiB/s wr, 517 op/s 2024-04-27T04:03:34.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:34 smithi046 ceph-mon[26899]: pgmap v245: 65 pgs: 65 active+clean; 77 MiB data, 456 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 6.7 MiB/s wr, 517 op/s 2024-04-27T04:03:36.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:36 smithi073 ceph-mon[34160]: pgmap v246: 65 pgs: 65 active+clean; 78 MiB data, 466 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 6.2 MiB/s wr, 517 op/s 2024-04-27T04:03:36.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:36 smithi046 ceph-mon[26899]: pgmap v246: 65 pgs: 65 active+clean; 78 MiB data, 466 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 6.2 MiB/s wr, 517 op/s 2024-04-27T04:03:38.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:38 smithi073 ceph-mon[34160]: pgmap v247: 65 pgs: 65 active+clean; 96 MiB data, 538 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 6.2 MiB/s wr, 572 op/s 2024-04-27T04:03:38.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:38 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:38.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:38 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:38.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:38 smithi046 ceph-mon[26899]: pgmap v247: 65 pgs: 65 active+clean; 96 MiB data, 538 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 6.2 MiB/s wr, 572 op/s 2024-04-27T04:03:38.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:38 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:38.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:38 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:40.021 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:39 smithi073 ceph-mon[34160]: pgmap v248: 65 pgs: 65 active+clean; 106 MiB data, 573 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.6 MiB/s wr, 400 op/s 2024-04-27T04:03:40.021 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:40.021 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:39 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:40.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:39 smithi046 ceph-mon[26899]: pgmap v248: 65 pgs: 65 active+clean; 106 MiB data, 573 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.6 MiB/s wr, 400 op/s 2024-04-27T04:03:40.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:40.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:39 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:42.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:41 smithi073 ceph-mon[34160]: pgmap v249: 65 pgs: 65 active+clean; 121 MiB data, 645 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 5.1 MiB/s wr, 480 op/s 2024-04-27T04:03:42.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:41 smithi046 ceph-mon[26899]: pgmap v249: 65 pgs: 65 active+clean; 121 MiB data, 645 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 5.1 MiB/s wr, 480 op/s 2024-04-27T04:03:44.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:44.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:44.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:03:44.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:03:44.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:44.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:03:44.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mgr fail", "who": "smithi046.qrfdvr"}]: dispatch 2024-04-27T04:03:44.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: Activating manager daemon smithi073.awyllt 2024-04-27T04:03:44.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: osdmap e47: 6 total, 6 up, 6 in 2024-04-27T04:03:44.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "mgr fail", "who": "smithi046.qrfdvr"}]': finished 2024-04-27T04:03:44.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: mgrmap e21: smithi073.awyllt(active, starting, since 0.00684531s) 2024-04-27T04:03:44.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-04-27T04:03:44.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-04-27T04:03:44.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi046.mycakd"}]: dispatch 2024-04-27T04:03:44.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi073.zkoypj"}]: dispatch 2024-04-27T04:03:44.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi073.yryono"}]: dispatch 2024-04-27T04:03:44.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi046.qgmaql"}]: dispatch 2024-04-27T04:03:44.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mgr metadata", "who": "smithi073.awyllt", "id": "smithi073.awyllt"}]: dispatch 2024-04-27T04:03:44.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T04:03:44.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T04:03:44.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T04:03:44.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T04:03:44.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T04:03:44.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T04:03:44.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T04:03:44.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T04:03:44.675 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T04:03:44.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: Manager daemon smithi073.awyllt is now available 2024-04-27T04:03:44.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:03:44.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:03:44.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.awyllt/mirror_snapshot_schedule"}]: dispatch 2024-04-27T04:03:44.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 ' entity='mgr.smithi073.awyllt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.awyllt/mirror_snapshot_schedule"}]: dispatch 2024-04-27T04:03:44.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.awyllt/trash_purge_schedule"}]: dispatch 2024-04-27T04:03:44.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: from='mgr.24405 ' entity='mgr.smithi073.awyllt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.awyllt/trash_purge_schedule"}]: dispatch 2024-04-27T04:03:44.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: [27/Apr/2024:04:03:43] ENGINE Bus STARTING 2024-04-27T04:03:44.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: Deploying cephadm binary to smithi073 2024-04-27T04:03:44.676 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:44 smithi046 ceph-mon[26899]: Deploying cephadm binary to smithi046 2024-04-27T04:03:44.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:44.825 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:44.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T04:03:44.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T04:03:44.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' 2024-04-27T04:03:44.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:03:44.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd=[{"prefix": "mgr fail", "who": "smithi046.qrfdvr"}]: dispatch 2024-04-27T04:03:44.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: Activating manager daemon smithi073.awyllt 2024-04-27T04:03:44.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: osdmap e47: 6 total, 6 up, 6 in 2024-04-27T04:03:44.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.14217 172.21.15.46:0/3245661392' entity='mgr.smithi046.qrfdvr' cmd='[{"prefix": "mgr fail", "who": "smithi046.qrfdvr"}]': finished 2024-04-27T04:03:44.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: mgrmap e21: smithi073.awyllt(active, starting, since 0.00684531s) 2024-04-27T04:03:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-04-27T04:03:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-04-27T04:03:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi046.mycakd"}]: dispatch 2024-04-27T04:03:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi073.zkoypj"}]: dispatch 2024-04-27T04:03:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi073.yryono"}]: dispatch 2024-04-27T04:03:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi046.qgmaql"}]: dispatch 2024-04-27T04:03:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mgr metadata", "who": "smithi073.awyllt", "id": "smithi073.awyllt"}]: dispatch 2024-04-27T04:03:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T04:03:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T04:03:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T04:03:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T04:03:44.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T04:03:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T04:03:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T04:03:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T04:03:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T04:03:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: Manager daemon smithi073.awyllt is now available 2024-04-27T04:03:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T04:03:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T04:03:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.awyllt/mirror_snapshot_schedule"}]: dispatch 2024-04-27T04:03:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 ' entity='mgr.smithi073.awyllt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.awyllt/mirror_snapshot_schedule"}]: dispatch 2024-04-27T04:03:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 172.21.15.73:0/2566493788' entity='mgr.smithi073.awyllt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.awyllt/trash_purge_schedule"}]: dispatch 2024-04-27T04:03:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: from='mgr.24405 ' entity='mgr.smithi073.awyllt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.awyllt/trash_purge_schedule"}]: dispatch 2024-04-27T04:03:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: [27/Apr/2024:04:03:43] ENGINE Bus STARTING 2024-04-27T04:03:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: Deploying cephadm binary to smithi073 2024-04-27T04:03:44.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:44 smithi073 ceph-mon[34160]: Deploying cephadm binary to smithi046 2024-04-27T04:03:45.894 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:45 smithi046 ceph-mon[26899]: [27/Apr/2024:04:03:44] ENGINE Serving on http://172.21.15.73:8765 2024-04-27T04:03:45.895 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:45 smithi046 ceph-mon[26899]: [27/Apr/2024:04:03:44] ENGINE Serving on https://172.21.15.73:7150 2024-04-27T04:03:45.895 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:45 smithi046 ceph-mon[26899]: [27/Apr/2024:04:03:44] ENGINE Bus STARTED 2024-04-27T04:03:45.895 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:45 smithi046 ceph-mon[26899]: mgrmap e22: smithi073.awyllt(active, since 1.00944s) 2024-04-27T04:03:45.895 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:45 smithi046 ceph-mon[26899]: pgmap v3: 65 pgs: 65 active+clean; 154 MiB data, 764 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:03:46.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:45 smithi073 ceph-mon[34160]: [27/Apr/2024:04:03:44] ENGINE Serving on http://172.21.15.73:8765 2024-04-27T04:03:46.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:45 smithi073 ceph-mon[34160]: [27/Apr/2024:04:03:44] ENGINE Serving on https://172.21.15.73:7150 2024-04-27T04:03:46.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:45 smithi073 ceph-mon[34160]: [27/Apr/2024:04:03:44] ENGINE Bus STARTED 2024-04-27T04:03:46.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:45 smithi073 ceph-mon[34160]: mgrmap e22: smithi073.awyllt(active, since 1.00944s) 2024-04-27T04:03:46.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:45 smithi073 ceph-mon[34160]: pgmap v3: 65 pgs: 65 active+clean; 154 MiB data, 764 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:03:46.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:46 smithi046 ceph-mon[26899]: pgmap v4: 65 pgs: 65 active+clean; 154 MiB data, 764 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:03:47.074 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:46 smithi073 ceph-mon[34160]: pgmap v4: 65 pgs: 65 active+clean; 154 MiB data, 764 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:03:47.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:47 smithi046 ceph-mon[26899]: mgrmap e23: smithi073.awyllt(active, since 3s) 2024-04-27T04:03:48.074 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:47 smithi073 ceph-mon[34160]: mgrmap e23: smithi073.awyllt(active, since 3s) 2024-04-27T04:03:48.102 INFO:tasks.workunit.client.1.smithi073.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-27T04:03:48.121 INFO:tasks.workunit.client.1.smithi073.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-27T04:03:48.122 INFO:tasks.workunit.client.1.smithi073.stderr:+ make 2024-04-27T04:03:48.687 INFO:tasks.workunit.client.1.smithi073.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-27T04:03:48.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:03:48 smithi046 ceph-mon[26899]: pgmap v5: 65 pgs: 65 active+clean; 154 MiB data, 764 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:03:49.015 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Apr 27 04:03:48 smithi073 ceph-mon[34160]: pgmap v5: 65 pgs: 65 active+clean; 154 MiB data, 764 MiB used, 536 GiB / 536 GiB avail 2024-04-27T04:03:57.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.725+0000 7fcffcb2b700 1 -- 172.21.15.46:0/1349652266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcff8102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T04:03:57.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.725+0000 7fcffcb2b700 1 --2- 172.21.15.46:0/1349652266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff8101eb0 0x7fcff8102310 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009910 tx=0x7fcfe0009c20 comp rx=0 tx=0).stop 2024-04-27T04:03:57.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.725+0000 7fcffcb2b700 1 -- 172.21.15.46:0/1349652266 shutdown_connections 2024-04-27T04:03:57.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.725+0000 7fcffcb2b700 1 --2- 172.21.15.46:0/1349652266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff8101eb0 0x7fcff8102310 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:57.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.725+0000 7fcffcb2b700 1 --2- 172.21.15.46:0/1349652266 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff80feed0 0x7fcff80ff2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:57.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.725+0000 7fcffcb2b700 1 -- 172.21.15.46:0/1349652266 >> 172.21.15.46:0/1349652266 conn(0x7fcff80fa830 msgr2=0x7fcff80fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T04:03:57.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.725+0000 7fcffcb2b700 1 -- 172.21.15.46:0/1349652266 shutdown_connections 2024-04-27T04:03:57.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.725+0000 7fcffcb2b700 1 -- 172.21.15.46:0/1349652266 wait complete. 2024-04-27T04:03:57.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.726+0000 7fcffcb2b700 1 Processor -- start 2024-04-27T04:03:57.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.726+0000 7fcffcb2b700 1 -- start start 2024-04-27T04:03:57.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.726+0000 7fcffcb2b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcff8196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:57.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.726+0000 7fcffcb2b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcff8196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:57.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.726+0000 7fcffcb2b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff8197260 con 0x7fcff80feed0 2024-04-27T04:03:57.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.727+0000 7fcffcb2b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff81973a0 con 0x7fcff8101eb0 2024-04-27T04:03:57.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.727+0000 7fcff659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcff8196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:03:57.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.727+0000 7fcff659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcff8196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35466/0 (socket says 172.21.15.46:35466) 2024-04-27T04:03:57.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.727+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 learned_addr learned my addr 172.21.15.46:0/1077475855 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-27T04:03:57.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.727+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcff8196c40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:03:57.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.727+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcff8196c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:03:57.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.727+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe00095f0 con 0x7fcff80feed0 2024-04-27T04:03:57.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.727+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcff8196700 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800b9e0 tx=0x7fcfe800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:03:57.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.727+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800c980 con 0x7fcff80feed0 2024-04-27T04:03:57.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.728+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8014440 con 0x7fcff80feed0 2024-04-27T04:03:57.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.728+0000 7fcffcb2b700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff819be00 con 0x7fcff80feed0 2024-04-27T04:03:57.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.728+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8012530 con 0x7fcff80feed0 2024-04-27T04:03:57.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.728+0000 7fcffcb2b700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff819c350 con 0x7fcff80feed0 2024-04-27T04:03:57.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.729+0000 7fcffcb2b700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff8190830 con 0x7fcff80feed0 2024-04-27T04:03:57.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.729+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe80126d0 con 0x7fcff80feed0 2024-04-27T04:03:57.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.729+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:03:57.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.729+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fcfe804d390 con 0x7fcff80feed0 2024-04-27T04:03:57.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.736+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188226 (secure 0 0 0) 0x7fcfe801b2c0 con 0x7fcff80feed0 2024-04-27T04:03:57.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:03:57.970+0000 7fcffcb2b700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcff802ce80 con 0x7fcfe40398c0 2024-04-27T04:04:07.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:07.738+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.73:6828/1981832083, fault. 2024-04-27T04:04:07.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:07.739+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-27T04:04:10.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:10.589+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:10.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:10.589+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:10.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:10.589+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:11.845 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:11.608+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:11.845 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:11.608+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:11.845 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:11.608+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:12.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:12.583+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:12.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:12.583+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:12.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:12.583+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:13.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.454+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcff8196700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:04:13.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.454+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcff8196700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:04:13.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.454+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcff8196700 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800b9e0 tx=0x7fcfe800bda0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:04:13.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.454+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcff8196700 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800b9e0 tx=0x7fcfe800bda0 comp rx=0 tx=0).stop 2024-04-27T04:04:13.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.454+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcff8196700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:04:13.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.454+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcff8196700 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:13.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.455+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe4043690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:13.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.455+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4046000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:13.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.455+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4046a10 con 0x7fcfe40412a0 2024-04-27T04:04:13.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.455+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4046b50 con 0x7fcff8101eb0 2024-04-27T04:04:13.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.455+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe4043690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:04:13.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.456+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4046000 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:04:13.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.456+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4046000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:13.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.456+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4046d40 con 0x7fcfe40412a0 2024-04-27T04:04:13.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.456+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe4043690 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804b780 tx=0x7fcfe8009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:04:13.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8014620 con 0x7fcfe40412a0 2024-04-27T04:04:13.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.457+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804d0f0 con 0x7fcfe40412a0 2024-04-27T04:04:13.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:13.458+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8012980 con 0x7fcfe40412a0 2024-04-27T04:04:13.508 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:13.245+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:13.508 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:13.245+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:13.508 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:13.245+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:13.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:13.538+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:13.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:13.538+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:13.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:13.538+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:13.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:13.617+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:13.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:13.617+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:13.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:13.617+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:14.519 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:14.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:14.519 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:14.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:14.519 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:14.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:14.519 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:14.268+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:14.519 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:14.268+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:14.519 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:14.268+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:14.773 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:14.665+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:14.773 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:14.665+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:14.773 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:14.665+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:15.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:15.476+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:15.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:15.476+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:15.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:15.476+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:15.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:15.237+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:15.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:15.237+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:15.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:15.237+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:16.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:15.684+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:16.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:15.684+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:16.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:15.684+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:16.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:16.256+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:16.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:16.256+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:16.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:16.256+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:16.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:16.448+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:16.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:16.448+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:16.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:16.448+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:17.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:16.674+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:17.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:16.674+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:17.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:16.674+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:17.668 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:17.494+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:17.669 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:17.494+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:17.669 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:17.494+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:17.669 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:17.270+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:17.669 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:17.270+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:17.669 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:17.270+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:17.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:17.668+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:17.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:17.668+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:17.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:17.668+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:17.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:17.950+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.73:6828/1981832083, fault. 2024-04-27T04:04:17.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:17.950+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-27T04:04:18.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.455+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe4043690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-27T04:04:18.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.455+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe4043690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:04:18.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.455+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe4043690 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804b780 tx=0x7fcfe8009760 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:04:18.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.455+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe4043690 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804b780 tx=0x7fcfe8009760 comp rx=0 tx=0).stop 2024-04-27T04:04:18.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.455+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe4043690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:04:18.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.455+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe4043690 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:18.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.455+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe404d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:18.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.455+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4046000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:18.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.455+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e120 con 0x7fcff80feed0 2024-04-27T04:04:18.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.455+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e260 con 0x7fcff8101eb0 2024-04-27T04:04:18.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.455+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe404d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:04:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.456+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4046000 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:04:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.456+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4046000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.456+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe404e3a0 con 0x7fcff80feed0 2024-04-27T04:04:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.456+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe404d6e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803a2e0 tx=0x7fcfe804f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:04:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8046480 con 0x7fcff80feed0 2024-04-27T04:04:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f8a0 con 0x7fcff80feed0 2024-04-27T04:04:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:18.457+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe800fba0 con 0x7fcff80feed0 2024-04-27T04:04:18.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:18.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:18.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:18.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:18.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:18.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:18.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:18.248+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:18.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:18.248+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:18.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:18.248+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:19.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:18.672+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:19.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:18.672+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:19.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:18.672+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:19.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:19.285+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:19.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:19.285+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:19.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:19.285+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:19.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:19.439+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:19.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:19.439+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:19.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:19.439+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:20.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:19.684+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:20.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:19.684+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:20.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:19.684+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:20.648 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:20.264+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:20.648 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:20.264+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:20.648 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:20.264+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:20.648 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:20.422+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:20.648 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:20.422+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:20.648 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:20.422+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:20.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:20.648+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:20.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:20.648+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:20.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:20.648+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:21.570 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:21.394+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:21.570 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:21.394+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:21.570 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:21.394+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:21.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:21.243+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:21.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:21.243+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:21.571 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:21.243+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:21.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:21.634+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:21.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:21.634+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:21.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:21.634+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:22.596 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:22.195+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:22.596 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:22.195+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:22.596 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:22.195+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:22.596 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:22.423+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:22.596 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:22.423+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:22.597 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:22.423+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:22.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:22.596+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:22.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:22.596+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:22.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:22.596+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:23.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.455+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcfe404d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-27T04:04:23.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.455+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcfe404d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:04:23.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.455+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe404d6e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803a2e0 tx=0x7fcfe804f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:04:23.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.455+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe404d6e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803a2e0 tx=0x7fcfe804f040 comp rx=0 tx=0).stop 2024-04-27T04:04:23.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.455+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcfe404d6e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:04:23.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.455+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe404d6e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:23.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.455+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4046000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:23.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.455+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:23.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.455+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404eb20 con 0x7fcfe40412a0 2024-04-27T04:04:23.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.455+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404ec60 con 0x7fcff8101eb0 2024-04-27T04:04:23.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.455+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:04:23.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.456+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4046000 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:04:23.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.456+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4046000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:23.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.456+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe404eda0 con 0x7fcfe40412a0 2024-04-27T04:04:23.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.457+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404e530 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4040de0 tx=0x7fcfe800b5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:04:23.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.457+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0011610 con 0x7fcfe40412a0 2024-04-27T04:04:23.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.457+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0009150 con 0x7fcfe40412a0 2024-04-27T04:04:23.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:23.458+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0010a20 con 0x7fcfe40412a0 2024-04-27T04:04:23.459 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:23.207+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:23.459 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:23.207+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:23.459 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:23.207+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:23.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:23.456+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:23.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:23.456+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:23.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:23.456+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:23.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:23.549+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:23.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:23.549+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:23.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:23.549+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:24.158+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:24.158+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:24.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:24.158+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:24.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:24.472+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:24.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:24.472+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:24.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:24.472+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:24.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:24.585+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:24.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:24.585+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:24.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:24.585+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:25.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:25.125+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:25.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:25.125+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:25.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:25.125+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:25.812 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:25.430+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:25.812 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:25.430+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:25.812 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:25.430+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:25.812 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:25.623+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:25.813 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:25.623+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:25.813 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:25.623+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:26.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:26.390+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:26.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:26.390+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:26.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:26.390+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:26.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:26.152+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:26.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:26.152+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:26.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:26.152+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:26.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:26.591+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:26.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:26.591+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:26.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:26.591+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:27.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:27.149+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:27.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:27.149+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:27.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:27.149+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:27.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:27.600+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:27.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:27.600+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:27.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:27.600+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:27.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:27.426+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:27.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:27.426+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:27.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:27.426+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:28.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.356+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.73:6828/1981832083, fault. 2024-04-27T04:04:28.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.356+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-27T04:04:28.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:28.145+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:28.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:28.145+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:28.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:28.145+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:28.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:28.380+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:28.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:28.380+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:28.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:28.380+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:28.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.455+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-27T04:04:28.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.455+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:04:28.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.455+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404e530 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4040de0 tx=0x7fcfe800b5f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:04:28.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.455+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404e530 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4040de0 tx=0x7fcfe800b5f0 comp rx=0 tx=0).stop 2024-04-27T04:04:28.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.455+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404e530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:04:28.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.455+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404e530 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:28.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.455+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4043ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:28.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.455+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:28.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.455+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f730 con 0x7fcff80feed0 2024-04-27T04:04:28.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.455+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f870 con 0x7fcff8101eb0 2024-04-27T04:04:28.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.456+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe404ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:04:28.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.457+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4043ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:04:28.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.457+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4043ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:28.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.457+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe404f9b0 con 0x7fcff80feed0 2024-04-27T04:04:28.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.457+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe404ef30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4041100 tx=0x7fcff0006e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:04:28.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.457+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff000aee0 con 0x7fcff80feed0 2024-04-27T04:04:28.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.458+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0010860 con 0x7fcff80feed0 2024-04-27T04:04:28.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:28.459+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0011770 con 0x7fcff80feed0 2024-04-27T04:04:28.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:28.588+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:28.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:28.588+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:28.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:28.588+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:29.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:29.142+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:29.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:29.142+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:29.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:29.142+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:29.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:29.560+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:29.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:29.560+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:29.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:29.560+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:29.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:29.427+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:29.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:29.427+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:29.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:29.427+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:30.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:30.107+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:30.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:30.107+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:30.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:30.107+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:30.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:30.575+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:30.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:30.575+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:30.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:30.575+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:30.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:30.471+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:30.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:30.471+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:30.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:30.471+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:31.415 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:31.105+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:31.415 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:31.105+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:31.415 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:31.105+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:31.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:31.504+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:31.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:31.504+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:31.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:31.504+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:31.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:31.549+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:31.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:31.549+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:31.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:31.549+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:32.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:32.083+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:32.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:32.083+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:32.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:32.083+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:32.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:32.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:32.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:32.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:32.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:32.562+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:32.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:32.547+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:32.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:32.547+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:32.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:32.547+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:33.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:33.116+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:33.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:33.116+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:33.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:33.116+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.455+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:04:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.455+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:04:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.455+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe404ef30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4041100 tx=0x7fcff0006e50 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:04:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.455+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe404ef30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4041100 tx=0x7fcff0006e50 comp rx=0 tx=0).stop 2024-04-27T04:04:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:04:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe404ef30 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4043ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404fd80 con 0x7fcfe40412a0 2024-04-27T04:04:33.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404fec0 con 0x7fcff8101eb0 2024-04-27T04:04:33.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.456+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:04:33.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.456+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4043ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:04:33.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.456+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4043ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:33.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.457+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4050000 con 0x7fcfe40412a0 2024-04-27T04:04:33.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.457+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404e530 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4018e30 tx=0x7fcff0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:04:33.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.457+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe804fa80 con 0x7fcfe40412a0 2024-04-27T04:04:33.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.457+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe801d540 con 0x7fcfe40412a0 2024-04-27T04:04:33.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:33.458+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe803a380 con 0x7fcfe40412a0 2024-04-27T04:04:33.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:33.593+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:33.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:33.593+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:33.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:33.593+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:33.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:33.596+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:33.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:33.596+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:33.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:33.596+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:34.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:34.164+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:34.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:34.164+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:34.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:34.164+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:34.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:34.640+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:34.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:34.640+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:34.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:34.640+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:34.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:34.626+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:34.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:34.626+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:34.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:34.626+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:35.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:35.125+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:35.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:35.125+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:35.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:35.125+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:35.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:35.661+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:35.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:35.661+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:35.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:35.661+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:35.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:35.579+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:35.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:35.579+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:35.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:35.579+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:36.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:36.133+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:36.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:36.133+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:36.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:36.133+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:36.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:36.619+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:36.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:36.619+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:36.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:36.619+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:36.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:36.630+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:36.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:36.630+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:36.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:36.630+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:37.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:37.139+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:37.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:37.139+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:37.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:37.139+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:37.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:37.654+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:37.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:37.654+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:37.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:37.654+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:37.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:37.626+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:37.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:37.626+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:37.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:37.626+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:38.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:38.137+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:38.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:38.137+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:38.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:38.137+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:38.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.456+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-27T04:04:38.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.456+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:04:38.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.456+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404e530 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4018e30 tx=0x7fcff0011000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:04:38.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.456+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404e530 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4018e30 tx=0x7fcff0011000 comp rx=0 tx=0).stop 2024-04-27T04:04:38.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404e530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:04:38.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404e530 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:38.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:38.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:38.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40500b0 con 0x7fcff80feed0 2024-04-27T04:04:38.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40501f0 con 0x7fcff8101eb0 2024-04-27T04:04:38.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.456+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:04:38.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.457+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:04:38.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.457+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:38.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.457+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4050330 con 0x7fcff80feed0 2024-04-27T04:04:38.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.457+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcff004bb40 tx=0x7fcfe8014d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:04:38.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.458+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0043080 con 0x7fcff80feed0 2024-04-27T04:04:38.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.459+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0011e50 con 0x7fcff80feed0 2024-04-27T04:04:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:38.460+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0001a30 con 0x7fcff80feed0 2024-04-27T04:04:38.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:38.599+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:38.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:38.599+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:38.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:38.599+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:38.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:38.660+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:38.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:38.660+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:38.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:38.660+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:39.167+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.73:6828/1981832083, fault. 2024-04-27T04:04:39.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:39.167+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-27T04:04:39.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:39.159+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:39.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:39.159+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:39.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:39.159+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:39.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:39.594+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:39.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:39.594+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:39.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:39.594+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:39.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:39.695+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:39.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:39.695+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:39.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:39.695+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:40.613 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:40.203+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:40.613 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:40.203+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:40.613 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:40.203+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:40.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:40.612+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:40.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:40.613+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:40.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:40.613+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:40.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:40.713+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:40.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:40.713+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:40.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:40.713+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:41.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:41.192+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:41.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:41.192+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:41.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:41.192+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:41.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:41.606+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:41.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:41.606+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:41.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:41.606+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:41.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:41.718+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:41.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:41.718+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:41.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:41.718+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:42.634 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:42.208+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:42.634 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:42.208+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:42.634 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:42.208+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:42.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:42.634+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:42.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:42.634+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:42.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:42.634+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:42.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:42.673+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:42.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:42.673+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:42.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:42.673+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:43.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.456+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcfe4046000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-27T04:04:43.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.456+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcfe4046000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:04:43.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.456+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcff004bb40 tx=0x7fcfe8014d20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:04:43.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.456+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcff004bb40 tx=0x7fcfe8014d20 comp rx=0 tx=0).stop 2024-04-27T04:04:43.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcfe4046000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:04:43.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:43.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:43.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:43.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4050a00 con 0x7fcfe40412a0 2024-04-27T04:04:43.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4050b40 con 0x7fcff8101eb0 2024-04-27T04:04:43.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.457+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:04:43.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.458+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:04:43.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.458+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:43.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.458+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4050c80 con 0x7fcfe40412a0 2024-04-27T04:04:43.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.458+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40349d0 tx=0x7fcff0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:04:43.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.459+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0011ac0 con 0x7fcfe40412a0 2024-04-27T04:04:43.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.459+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff003a230 con 0x7fcfe40412a0 2024-04-27T04:04:43.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:43.460+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff000dad0 con 0x7fcfe40412a0 2024-04-27T04:04:43.510 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:43.247+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:43.510 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:43.247+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:43.510 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:43.247+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:43.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:43.704+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:43.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:43.704+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:43.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:43.704+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:43.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:43.665+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:43.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:43.665+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:43.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:43.665+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:44.623 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:44.209+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:44.623 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:44.209+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:44.623 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:44.209+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:44.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:44.743+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:44.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:44.743+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:44.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:44.743+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:44.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:44.623+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:44.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:44.623+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:44.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:44.623+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:45.621 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:45.172+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:45.621 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:45.172+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:45.621 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:45.172+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:45.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:45.698+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:45.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:45.698+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:45.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:45.698+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:45.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:45.621+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:45.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:45.621+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:45.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:45.621+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:46.578 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:46.173+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:46.579 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:46.173+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:46.579 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:46.173+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:46.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:46.702+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:46.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:46.702+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:46.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:46.702+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:46.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:46.649+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:46.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:46.649+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:46.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:46.649+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:47.634 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:47.195+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:47.634 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:47.195+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:47.634 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:47.195+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:47.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:47.633+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:47.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:47.633+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:47.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:47.633+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:47.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:47.731+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:47.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:47.731+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:47.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:47.731+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:48.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:48.156+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:48.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:48.156+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:48.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:48.156+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:48.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.455+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:04:48.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.455+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:04:48.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.455+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40349d0 tx=0x7fcff0000bc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:04:48.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.455+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40349d0 tx=0x7fcff0000bc0 comp rx=0 tx=0).stop 2024-04-27T04:04:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:04:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4050ec0 con 0x7fcff80feed0 2024-04-27T04:04:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051000 con 0x7fcff8101eb0 2024-04-27T04:04:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.456+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:04:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.456+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:04:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.456+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.456+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4051140 con 0x7fcff80feed0 2024-04-27T04:04:48.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.457+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8016fa0 tx=0x7fcff0010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:04:48.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.459+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800cea0 con 0x7fcff80feed0 2024-04-27T04:04:48.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.459+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804fbe0 con 0x7fcff80feed0 2024-04-27T04:04:48.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:48.460+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013a90 con 0x7fcff80feed0 2024-04-27T04:04:48.739 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:48.651+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:48.739 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:48.651+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:48.740 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:48.651+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:49.114 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:48.738+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:49.114 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:48.738+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:49.114 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:48.738+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:49.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:49.113+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:49.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:49.113+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:49.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:49.113+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:50.065 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:49.697+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:50.066 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:49.697+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:50.066 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:49.697+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:50.066 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:49.782+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:50.066 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:49.782+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:50.066 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:49.782+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:50.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:50.064+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:50.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:50.064+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:50.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:50.064+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:50.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:50.771+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.73:6828/1981832083, fault. 2024-04-27T04:04:50.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:50.771+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-27T04:04:50.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:50.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:50.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:50.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:50.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:50.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:50.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:50.662+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:50.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:50.662+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:50.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:50.662+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:51.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:51.068+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:51.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:51.068+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:51.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:51.068+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:51.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:51.666+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:51.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:51.666+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:51.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:51.666+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:51.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:51.815+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:51.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:51.815+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:51.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:51.815+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:52.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:52.057+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:52.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:52.057+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:52.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:52.057+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:52.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:52.647+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:52.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:52.647+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:52.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:52.647+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:52.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:52.798+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:52.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:52.798+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:52.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:52.798+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:53.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:53.100+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:53.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:53.100+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:53.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:53.100+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:53.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.456+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcfe4046000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:04:53.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.456+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcfe4046000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:04:53.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.456+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8016fa0 tx=0x7fcff0010f00 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:04:53.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.456+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8016fa0 tx=0x7fcff0010f00 comp rx=0 tx=0).stop 2024-04-27T04:04:53.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcfe4046000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:04:53.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:53.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:53.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:53.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051020 con 0x7fcfe40412a0 2024-04-27T04:04:53.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051160 con 0x7fcff8101eb0 2024-04-27T04:04:53.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.456+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:04:53.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.457+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:04:53.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.457+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:53.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.457+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40512a0 con 0x7fcfe40412a0 2024-04-27T04:04:53.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.457+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8054c20 tx=0x7fcfe800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:04:53.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.459+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800cea0 con 0x7fcfe40412a0 2024-04-27T04:04:53.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.460+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe801b8c0 con 0x7fcfe40412a0 2024-04-27T04:04:53.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:53.461+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe803a6b0 con 0x7fcfe40412a0 2024-04-27T04:04:53.820 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:53.635+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:53.820 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:53.635+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:53.820 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:53.635+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:54.147 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:53.819+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:54.147 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:53.819+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:54.147 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:53.819+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:54.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:54.146+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:54.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:54.146+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:54.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:54.146+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:54.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:54.816+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:54.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:54.817+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:54.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:54.817+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:54.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:54.600+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:54.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:54.600+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:54.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:54.600+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:55.128+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:55.128+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:55.128+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:55.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:55.865+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:55.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:55.865+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:55.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:55.865+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:55.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:55.629+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:55.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:55.629+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:55.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:55.629+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:56.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:56.096+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:56.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:56.096+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:56.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:56.096+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:56.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:56.852+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:04:56.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:56.852+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-27T04:04:56.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:56.897+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:56.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:56.897+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:56.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:56.897+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:56.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:56.652+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:56.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:56.652+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:56.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:56.652+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:57.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:57.080+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:57.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:57.080+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:57.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:57.080+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:57.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:57.891+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:57.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:57.891+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:57.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:57.891+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:57.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:57.657+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:57.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:57.657+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:57.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:57.657+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:58.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:58.080+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:58.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:58.080+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:58.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:58.080+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:58.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.456+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:04:58.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.456+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:04:58.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.456+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8054c20 tx=0x7fcfe800bda0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:04:58.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.456+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8054c20 tx=0x7fcfe800bda0 comp rx=0 tx=0).stop 2024-04-27T04:04:58.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:04:58.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:58.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:58.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.456+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:04:58.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051020 con 0x7fcff80feed0 2024-04-27T04:04:58.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.456+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051160 con 0x7fcff8101eb0 2024-04-27T04:04:58.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.457+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:04:58.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.457+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:04:58.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.458+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:04:58.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.458+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40512a0 con 0x7fcff80feed0 2024-04-27T04:04:58.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.458+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8056070 tx=0x7fcfe8014d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:04:58.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.459+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe803aa70 con 0x7fcff80feed0 2024-04-27T04:04:58.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.460+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe80469b0 con 0x7fcff80feed0 2024-04-27T04:04:58.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:04:58.460+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8042ca0 con 0x7fcff80feed0 2024-04-27T04:04:58.900 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:58.618+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:58.900 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:58.618+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:58.900 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:58.618+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:59.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:58.898+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:59.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:58.898+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:59.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:58.898+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:04:59.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:59.054+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:59.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:59.054+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:59.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:04:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:04:59.054+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:04:59.865 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:59.569+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:59.865 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:59.569+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:04:59.865 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:04:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:04:59.569+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:00.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:59.864+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:00.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:59.864+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:00.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:04:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:04:59.864+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:00.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:00.025+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:00.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:00.025+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:00.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:00.025+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:00.870 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:00.609+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:00.870 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:00.609+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:00.871 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:00.609+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:01.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:00.870+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:01.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:00.870+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:01.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:00.870+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:01.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:00.983+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:01.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:00.983+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:01.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:00.983+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:01.888 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:01.634+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:01.888 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:01.634+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:01.888 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:01.634+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:02.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:01.999+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:02.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:01.999+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:02.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:01.999+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:02.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:01.888+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:02.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:01.888+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:02.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:01.888+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:02.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:02.669+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:02.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:02.669+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:02.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:02.669+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:03.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:02.923+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:03.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:02.923+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:03.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:02.923+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:03.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:03.013+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:03.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:03.013+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:03.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:03.013+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:03.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.457+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcfe4046000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:05:03.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.457+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcfe4046000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:05:03.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.457+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8056070 tx=0x7fcfe8014d20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:05:03.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.457+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8056070 tx=0x7fcfe8014d20 comp rx=0 tx=0).stop 2024-04-27T04:05:03.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.457+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 msgr2=0x7fcfe4046000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:05:03.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.457+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcff80feed0 0x7fcfe4046000 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:03.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.457+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:03.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.457+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40505a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:03.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.457+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40511b0 con 0x7fcfe40412a0 2024-04-27T04:05:03.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.457+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40512f0 con 0x7fcff8101eb0 2024-04-27T04:05:03.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.458+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:05:03.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.459+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:05:03.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.459+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:03.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.459+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4051430 con 0x7fcfe40412a0 2024-04-27T04:05:03.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.459+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40505a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fcff0017e60 tx=0x7fcfe8013f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:05:03.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.460+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0009e80 con 0x7fcfe40412a0 2024-04-27T04:05:03.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.461+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0009e80 con 0x7fcfe40412a0 2024-04-27T04:05:03.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:03.461+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe40412a0 2024-04-27T04:05:03.880 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:03.651+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:03.880 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:03.651+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:03.880 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:03.651+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:04.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:03.991+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:04.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:03.991+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:04.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:03.991+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:04.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:03.880+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:04.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:03.880+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:04.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:03.880+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:04.898 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:04.608+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:04.898 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:04.608+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:04.898 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:04.608+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:05.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:04.987+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:05.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:04.987+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:05.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:04.987+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:05.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:04.898+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:05.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:04.898+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:05.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:04.898+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:05.909 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:05.583+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:05.909 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:05.583+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:05.909 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:05.583+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:06.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:05.974+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:06.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:05.974+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:06.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:05.974+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:06.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:05.909+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:06.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:05.909+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:06.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:05.909+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:06.886 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:06.555+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:06.886 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:06.555+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:06.886 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:06.555+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:07.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:06.992+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:07.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:06.992+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:07.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:06.992+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:07.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:06.885+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:07.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:06.885+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:07.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:06.885+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:07.910 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:07.556+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:07.911 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:07.556+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:07.911 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:07.556+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:08.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:07.911+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:08.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:07.911+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:08.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:07.911+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:08.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:08.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:08.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:08.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:08.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:08.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:08.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.457+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40505a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-27T04:05:08.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.457+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40505a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:05:08.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.457+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40505a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fcff0017e60 tx=0x7fcfe8013f20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:05:08.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.457+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40505a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fcff0017e60 tx=0x7fcfe8013f20 comp rx=0 tx=0).stop 2024-04-27T04:05:08.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.457+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40505a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:05:08.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.457+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40505a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:08.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.458+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4043ff0 0x7fcfe40515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:08.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.458+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:08.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.458+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051280 con 0x7fcfe4043ff0 2024-04-27T04:05:08.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.458+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051b00 con 0x7fcff8101eb0 2024-04-27T04:05:08.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.458+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4043ff0 0x7fcfe40515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:05:08.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.459+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:05:08.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.459+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:08.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.459+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4051c40 con 0x7fcfe4043ff0 2024-04-27T04:05:08.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.459+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4043ff0 0x7fcfe40515c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400fbe0 tx=0x7fcff004b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:05:08.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.460+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe801e620 con 0x7fcfe4043ff0 2024-04-27T04:05:08.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.460+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe801e440 con 0x7fcfe4043ff0 2024-04-27T04:05:08.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:08.461+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe801ea10 con 0x7fcfe4043ff0 2024-04-27T04:05:08.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:08.598+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:08.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:08.598+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:08.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:08.598+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:09.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:09.141+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:05:09.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:09.141+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-27T04:05:09.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:08.957+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:09.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:08.957+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:09.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:08.957+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:09.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:09.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:09.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:09.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:09.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:09.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:09.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:09.599+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:09.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:09.599+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:09.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:09.599+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:10.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:09.957+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:10.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:09.957+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:10.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:09.957+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:10.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:09.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:10.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:09.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:10.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:09.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:10.910 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:10.603+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:10.910 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:10.603+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:10.910 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:10.603+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:11.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:10.910+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:11.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:10.910+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:11.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:10.910+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:11.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:10.935+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:11.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:10.935+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:11.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:10.935+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:11.844 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:11.648+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:11.844 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:11.648+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:11.844 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:11.648+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:11.910+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:11.910+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:11.910+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:12.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:11.864+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:12.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:11.864+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:12.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:11.864+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:12.892 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:12.636+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:12.892 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:12.637+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:12.892 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:12.637+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:12.892 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:12.874+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:12.893 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:12.874+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:12.893 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:12.874+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:13.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:12.892+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:13.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:12.892+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:13.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:12.892+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.458+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4043ff0 msgr2=0x7fcfe40515c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-27T04:05:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.458+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4043ff0 msgr2=0x7fcfe40515c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:05:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.458+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4043ff0 0x7fcfe40515c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400fbe0 tx=0x7fcff004b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:05:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.458+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4043ff0 0x7fcfe40515c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400fbe0 tx=0x7fcff004b040 comp rx=0 tx=0).stop 2024-04-27T04:05:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.458+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4043ff0 msgr2=0x7fcfe40515c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:05:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.458+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4043ff0 0x7fcfe40515c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.458+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.458+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40505a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.458+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051b00 con 0x7fcfe40412a0 2024-04-27T04:05:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.458+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051c40 con 0x7fcff8101eb0 2024-04-27T04:05:13.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.459+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:05:13.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.460+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:05:13.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.460+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.460+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4051d80 con 0x7fcfe40412a0 2024-04-27T04:05:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.460+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40505a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcff004eb00 tx=0x7fcfe8014900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:05:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.460+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003a230 con 0x7fcfe40412a0 2024-04-27T04:05:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.461+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff003a230 con 0x7fcfe40412a0 2024-04-27T04:05:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:13.462+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0001510 con 0x7fcfe40412a0 2024-04-27T04:05:13.861 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:13.634+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:13.861 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:13.634+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:13.861 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:13.634+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:14.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:13.876+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:14.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:13.876+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:14.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:13.876+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:14.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:13.861+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:14.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:13.861+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:14.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:13.861+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:14.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:14.642+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:14.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:14.642+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:14.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:14.642+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:14.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:14.875+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:14.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:14.875+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:14.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:14.875+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:14.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:14.838+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:14.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:14.838+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:14.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:14.838+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:15.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:15.874+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:15.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:15.874+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:15.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:15.874+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:15.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:15.628+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:15.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:15.628+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:15.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:15.628+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:15.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:15.829+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:15.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:15.829+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:15.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:15.829+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:16.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:16.658+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:16.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:16.658+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:16.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:16.658+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:16.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:16.884+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:16.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:16.884+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:16.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:16.884+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:16.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:16.859+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:16.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:16.859+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:16.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:16.859+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:17.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:17.654+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:17.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:17.654+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:17.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:17.654+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:17.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:17.857+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:17.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:17.857+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:17.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:17.857+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:17.926+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:17.926+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:17.926+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:18.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.458+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40505a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:05:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.458+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40505a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:05:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.458+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40505a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcff004eb00 tx=0x7fcfe8014900 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:05:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.458+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40505a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcff004eb00 tx=0x7fcfe8014900 comp rx=0 tx=0).stop 2024-04-27T04:05:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.458+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40505a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:05:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.458+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40505a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.458+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40515c0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.458+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.458+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051a70 con 0x7fcfe40515c0 2024-04-27T04:05:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.458+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051bb0 con 0x7fcff8101eb0 2024-04-27T04:05:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.459+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40515c0 0x7fcfe4051f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:05:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.459+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:05:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.459+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.459+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052450 con 0x7fcfe40515c0 2024-04-27T04:05:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.459+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40515c0 0x7fcfe4051f10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8033e30 tx=0x7fcff0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:05:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.461+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe801e440 con 0x7fcfe40515c0 2024-04-27T04:05:18.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.461+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8033e90 con 0x7fcfe40515c0 2024-04-27T04:05:18.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:18.462+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe801ea10 con 0x7fcfe40515c0 2024-04-27T04:05:18.874 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:18.664+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:18.874 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:18.664+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:18.874 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:18.664+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:19.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:18.874+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:19.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:18.874+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:19.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:18.874+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:19.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:18.913+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:19.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:18.913+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:19.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:18.913+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:19.951 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:19.838+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:19.951 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:19.838+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:19.951 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:19.838+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:19.952 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:19.691+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:19.952 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:19.691+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:19.952 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:19.691+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:19.951+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:19.951+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:19.951+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:20.935 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:20.682+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:20.936 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:20.682+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:20.936 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:20.682+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:20.936 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:20.845+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:20.936 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:20.845+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:20.936 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:20.845+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:21.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:20.935+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:21.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:20.935+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:21.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:20.935+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:21.844 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:21.705+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:21.844 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:21.705+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:21.844 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:21.705+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:22.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:21.942+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:22.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:21.942+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:22.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:21.942+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:22.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:21.844+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:22.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:21.844+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:22.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:21.844+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:23.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:22.808+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:23.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:22.808+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:23.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:22.808+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:23.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:22.721+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:23.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:22.721+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:23.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:22.721+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:23.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:22.953+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:23.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:22.953+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:23.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:22.953+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:23.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.458+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40515c0 msgr2=0x7fcfe4051f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:05:23.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.458+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40515c0 msgr2=0x7fcfe4051f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:05:23.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.458+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40515c0 0x7fcfe4051f10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8033e30 tx=0x7fcff0000bc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:05:23.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.458+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40515c0 0x7fcfe4051f10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8033e30 tx=0x7fcff0000bc0 comp rx=0 tx=0).stop 2024-04-27T04:05:23.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.458+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40515c0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:05:23.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.458+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40515c0 0x7fcfe4051f10 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:23.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.458+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:23.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.458+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:23.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.459+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051b20 con 0x7fcfe40412a0 2024-04-27T04:05:23.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.459+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052450 con 0x7fcff8101eb0 2024-04-27T04:05:23.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.459+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:05:23.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.460+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:05:23.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.460+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:23.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.460+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052590 con 0x7fcfe40412a0 2024-04-27T04:05:23.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.460+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804a050 tx=0x7fcfe80089c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:05:23.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.460+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8033e90 con 0x7fcfe40412a0 2024-04-27T04:05:23.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.461+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8033e90 con 0x7fcfe40412a0 2024-04-27T04:05:23.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:23.462+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe40412a0 2024-04-27T04:05:24.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:23.797+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:24.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:23.797+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:24.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:23.797+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:24.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:23.768+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:24.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:23.768+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:24.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:23.768+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:24.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:23.922+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:24.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:23.922+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:24.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:23.922+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:24.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:24.501+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:05:24.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:24.501+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:05:25.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:24.799+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:25.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:24.799+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:25.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:24.799+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:25.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:24.801+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:25.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:24.801+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:25.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:24.801+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:25.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:24.878+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:25.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:24.878+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:25.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:24.878+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:26.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:25.873+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:26.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:25.873+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:26.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:25.873+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:26.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:25.793+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:26.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:25.793+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:26.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:25.793+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:26.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:25.844+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:26.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:25.844+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:26.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:25.844+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:26.999 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:26.902+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:27.000 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:26.902+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:27.000 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:26.902+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:27.000 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:26.797+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:27.000 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:26.797+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:27.000 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:26.797+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:27.000 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:26.748+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:27.000 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:26.748+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:27.001 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:26.748+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:27.832 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:27.752+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:27.832 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:27.752+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:27.832 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:27.752+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:28.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:27.867+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:28.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:27.867+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:28.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:27.867+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:28.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:27.832+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:28.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:27.832+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:28.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:27.832+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:28.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.459+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:05:28.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.459+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:05:28.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.459+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804a050 tx=0x7fcfe80089c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:05:28.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.459+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804a050 tx=0x7fcfe80089c0 comp rx=0 tx=0).stop 2024-04-27T04:05:28.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.459+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:05:28.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.459+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:28.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.459+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:28.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.459+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:28.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.459+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044dc0 con 0x7fcfe40443d0 2024-04-27T04:05:28.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.459+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044f00 con 0x7fcff8101eb0 2024-04-27T04:05:28.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.460+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:05:28.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.460+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:05:28.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.460+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:28.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.460+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4045040 con 0x7fcfe40443d0 2024-04-27T04:05:28.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.461+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8026420 tx=0x7fcfe80303e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:05:28.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.461+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8013a90 con 0x7fcfe40443d0 2024-04-27T04:05:28.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.462+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8043ad0 con 0x7fcfe40443d0 2024-04-27T04:05:28.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:28.462+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8043cc0 con 0x7fcfe40443d0 2024-04-27T04:05:29.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:28.856+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:29.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:28.856+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:29.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:28.856+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:29.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:28.917+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:29.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:28.917+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:29.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:28.917+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:29.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:28.733+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:29.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:28.733+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:29.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:28.733+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:30.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:29.948+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:30.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:29.948+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:30.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:29.948+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:30.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:29.877+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:30.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:29.878+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:30.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:29.878+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:30.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:29.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:30.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:29.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:30.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:29.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:30.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:30.891+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:30.943 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:30.891+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:30.943 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:30.891+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:30.943 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:30.774+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:30.943 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:30.774+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:30.944 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:30.774+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:31.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:30.945+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:31.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:30.946+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:31.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:30.946+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:32.054 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:31.866+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:32.054 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:31.866+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:32.054 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:31.866+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:32.054 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:31.802+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:32.055 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:31.802+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:32.055 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:31.802+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:32.055 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:31.951+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:32.055 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:31.951+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:32.055 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:31.951+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:33.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:32.917+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:33.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:32.917+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:33.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:32.917+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:33.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:32.903+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:33.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:32.903+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:33.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:32.903+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:33.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:32.797+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:33.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:32.797+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:33.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:32.797+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.460+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:05:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.460+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:05:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.460+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8026420 tx=0x7fcfe80303e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:05:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.460+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8026420 tx=0x7fcfe80303e0 comp rx=0 tx=0).stop 2024-04-27T04:05:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.460+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:05:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.460+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.460+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.460+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.460+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044da0 con 0x7fcfe40412a0 2024-04-27T04:05:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.460+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044ee0 con 0x7fcff8101eb0 2024-04-27T04:05:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.460+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:05:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.461+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:05:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.461+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.461+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4045020 con 0x7fcfe40412a0 2024-04-27T04:05:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.462+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803efc0 tx=0x7fcfe801c5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:05:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.462+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8013a90 con 0x7fcfe40412a0 2024-04-27T04:05:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.462+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8013740 con 0x7fcfe40412a0 2024-04-27T04:05:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:33.463+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013c60 con 0x7fcfe40412a0 2024-04-27T04:05:34.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:33.760+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:34.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:33.760+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:34.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:33.760+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:34.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:33.940+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:34.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:33.940+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:34.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:33.940+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:34.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:33.963+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:34.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:33.963+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:34.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:33.963+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:35.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:34.968+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:35.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:34.968+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:35.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:34.968+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:34.762+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:35.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:34.762+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:35.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:34.762+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:35.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:35.001+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:35.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:35.001+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:35.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:35.001+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:36.000 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:35.749+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:36.000 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:35.749+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:36.000 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:35.749+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:36.000 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:35.920+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:36.000 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:35.920+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:36.000 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:35.920+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:36.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:36.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:36.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:36.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:36.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:36.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:37.109 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:36.959+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:37.109 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:36.959+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:37.109 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:36.959+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:37.109 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:36.757+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:37.109 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:36.757+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:37.109 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:36.757+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:37.110 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:36.989+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:37.110 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:36.989+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:37.110 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:36.989+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:38.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:37.942+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:38.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:37.942+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:38.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:37.942+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:38.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:37.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:38.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:37.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:38.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:37.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:38.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:37.729+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:38.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:37.729+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:38.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:37.729+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.461+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:05:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.461+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:05:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.461+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803efc0 tx=0x7fcfe801c5e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:05:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.461+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803efc0 tx=0x7fcfe801c5e0 comp rx=0 tx=0).stop 2024-04-27T04:05:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.461+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:05:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.461+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.461+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.461+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.461+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcfe40443d0 2024-04-27T04:05:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.461+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044fd0 con 0x7fcff8101eb0 2024-04-27T04:05:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.461+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:05:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.462+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:05:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.462+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.462+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4045110 con 0x7fcfe40443d0 2024-04-27T04:05:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.462+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fcff0028fc0 tx=0x7fcfe800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:05:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.463+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0010e10 con 0x7fcfe40443d0 2024-04-27T04:05:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.463+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe40443d0 2024-04-27T04:05:38.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:38.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0005b20 con 0x7fcfe40443d0 2024-04-27T04:05:38.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:38.693+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:38.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:38.693+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:38.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:38.693+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:39.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:38.951+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:39.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:38.951+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:39.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:38.951+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:39.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:38.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:39.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:38.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:39.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:38.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:39.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:39.861+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:05:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:39.861+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:05:39.960 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:39.703+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:39.960 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:39.703+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:39.960 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:39.703+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:40.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:40.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:40.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:40.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:40.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:40.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:40.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:39.960+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:40.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:39.960+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:40.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:39.960+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:40.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:40.665+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:40.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:40.665+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:40.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:40.665+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:41.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:40.988+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:41.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:40.988+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:41.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:40.988+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:41.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:40.969+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:41.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:40.969+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:41.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:40.969+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:41.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:41.638+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:41.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:41.638+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:41.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:41.638+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:42.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:41.961+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:42.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:41.961+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:42.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:41.961+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:42.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:41.977+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:42.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:41.977+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:42.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:41.977+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:42.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:42.637+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:42.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:42.637+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:42.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:42.637+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:43.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:43.004+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:43.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:43.004+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:43.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:43.004+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:43.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:42.927+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:43.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:42.927+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:43.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:42.927+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:43.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.461+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:05:43.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.462+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:05:43.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.462+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fcff0028fc0 tx=0x7fcfe800bda0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:05:43.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.462+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fcff0028fc0 tx=0x7fcfe800bda0 comp rx=0 tx=0).stop 2024-04-27T04:05:43.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.462+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:05:43.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.462+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:43.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.462+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:43.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.462+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:43.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.462+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcfe40412a0 2024-04-27T04:05:43.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.462+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044fd0 con 0x7fcff8101eb0 2024-04-27T04:05:43.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.462+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:05:43.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.463+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:05:43.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.463+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:43.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.463+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40412a0 2024-04-27T04:05:43.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.463+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcff004df50 tx=0x7fcff0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:05:43.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0030070 con 0x7fcfe40412a0 2024-04-27T04:05:43.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff00462d0 con 0x7fcfe40412a0 2024-04-27T04:05:43.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:43.465+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff003a520 con 0x7fcfe40412a0 2024-04-27T04:05:43.911 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:43.589+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:43.911 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:43.589+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:43.911 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:43.589+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:44.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:43.911+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:44.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:43.911+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:44.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:43.911+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:44.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:44.007+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:44.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:44.007+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:44.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:44.007+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:44.865 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:44.605+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:44.865 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:44.605+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:44.865 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:44.605+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:45.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:44.990+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:45.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:44.990+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:45.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:44.990+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:45.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:44.865+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:45.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:44.865+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:45.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:44.865+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:45.913 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:45.565+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:45.913 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:45.565+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:45.913 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:45.565+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:46.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:45.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:46.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:45.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:46.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:45.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:46.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:46.014+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:46.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:46.014+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:46.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:46.014+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:46.910 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:46.556+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:46.910 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:46.556+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:46.911 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:46.556+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:47.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:46.910+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:47.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:46.910+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:47.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:46.910+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:47.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:46.974+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:47.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:46.974+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:47.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:46.974+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:47.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:47.559+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:47.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:47.559+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:47.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:47.559+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:48.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:47.953+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:48.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:47.953+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:48.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:47.953+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:48.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:48.012+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:48.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:48.012+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:48.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:48.012+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.462+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:05:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.462+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:05:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.462+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcff004df50 tx=0x7fcff0000bc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:05:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.462+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcff004df50 tx=0x7fcff0000bc0 comp rx=0 tx=0).stop 2024-04-27T04:05:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.462+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:05:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.462+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.462+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.462+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.462+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcfe40443d0 2024-04-27T04:05:48.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.462+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044fd0 con 0x7fcff8101eb0 2024-04-27T04:05:48.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.462+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:05:48.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.463+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:05:48.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.463+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe404ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:48.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.463+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40443d0 2024-04-27T04:05:48.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.463+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8057bf0 tx=0x7fcff0010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:05:48.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8013a90 con 0x7fcfe40443d0 2024-04-27T04:05:48.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe801be70 con 0x7fcfe40443d0 2024-04-27T04:05:48.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:48.465+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013c60 con 0x7fcfe40443d0 2024-04-27T04:05:48.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:48.558+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:48.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:48.558+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:48.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:48.558+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:49.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:48.944+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:49.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:48.944+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:49.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:48.944+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:49.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:48.983+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:49.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:48.983+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:49.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:48.983+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:49.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:49.574+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:49.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:49.574+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:49.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:49.574+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:50.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:49.958+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:50.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:49.958+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:50.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:49.958+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:50.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:49.972+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:50.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:49.972+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:50.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:49.972+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:50.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:50.550+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:50.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:50.550+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:50.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:50.550+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:51.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:50.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:51.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:50.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:51.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:50.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:51.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:50.979+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:51.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:50.979+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:51.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:50.979+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:51.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:51.598+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:51.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:51.598+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:51.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:51.598+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:52.280 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:51.999+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:52.280 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:51.999+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:52.280 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:51.999+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:52.281 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:52.023+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:52.281 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:52.023+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:52.281 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:52.023+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:52.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:52.612+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:52.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:52.612+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:52.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:52.612+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:53.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:52.983+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:53.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:52.983+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:53.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:52.983+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:53.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:53.026+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:53.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:53.026+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:53.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:53.026+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:53.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.462+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:05:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.462+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:05:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.462+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8057bf0 tx=0x7fcff0010d70 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:05:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.462+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8057bf0 tx=0x7fcff0010d70 comp rx=0 tx=0).stop 2024-04-27T04:05:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.462+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:05:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.462+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.462+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.462+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.462+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcfe40412a0 2024-04-27T04:05:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.462+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044fd0 con 0x7fcff8101eb0 2024-04-27T04:05:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.463+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:05:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.463+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:05:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.463+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40504c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.463+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40412a0 2024-04-27T04:05:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.464+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fcff004f870 tx=0x7fcfe8054430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:05:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0030070 con 0x7fcfe40412a0 2024-04-27T04:05:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.465+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0007ea0 con 0x7fcfe40412a0 2024-04-27T04:05:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:53.466+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff003a520 con 0x7fcfe40412a0 2024-04-27T04:05:53.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:05:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:05:53.462+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:05:53.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:53.614+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:53.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:53.614+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:53.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:53.614+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:53.945+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:53.945+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:53.945+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:54.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:54.013+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:54.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:54.014+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:54.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:54.014+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:54.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:54.580+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:54.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:54.580+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:54.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:54.580+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:55.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:55.221+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:05:55.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:55.221+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:05:55.221 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:54.959+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:55.221 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:54.959+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:55.221 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:54.959+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:55.221 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:54.981+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:55.222 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:54.981+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:55.222 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:54.981+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:55.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:55.554+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:55.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:55.554+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:55.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:55.554+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:56.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:55.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:56.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:55.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:56.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:55.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:56.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:56.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:56.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:56.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:56.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:56.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:56.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:56.545+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:56.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:56.545+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:56.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:56.545+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:57.336 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:56.926+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:57.336 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:56.926+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:57.336 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:56.926+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:57.336 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:57.019+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:57.337 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:57.019+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:57.337 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:57.019+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:57.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:57.580+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:57.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:57.580+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:57.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:57.580+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:58.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:57.958+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:58.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:57.958+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:58.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:57.958+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:58.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:58.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:58.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:58.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:58.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:58.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:58.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.463+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:05:58.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.463+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:05:58.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.463+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fcff004f870 tx=0x7fcfe8054430 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:05:58.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.463+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fcff004f870 tx=0x7fcfe8054430 comp rx=0 tx=0).stop 2024-04-27T04:05:58.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.463+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:05:58.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.463+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:58.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.463+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:58.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.463+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:05:58.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.463+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcfe40443d0 2024-04-27T04:05:58.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.463+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044fd0 con 0x7fcff8101eb0 2024-04-27T04:05:58.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.464+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:05:58.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.464+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:05:58.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.464+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:05:58.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.464+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40443d0 2024-04-27T04:05:58.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.464+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcff001c990 tx=0x7fcff0028000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:05:58.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0030040 con 0x7fcfe40443d0 2024-04-27T04:05:58.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0010e50 con 0x7fcfe40443d0 2024-04-27T04:05:58.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:05:58.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0041ce0 con 0x7fcfe40443d0 2024-04-27T04:05:58.919 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:05:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:05:58.463+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:05:58.919 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:58.599+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:58.919 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:58.599+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:58.919 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:58.599+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:59.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:59.081+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:59.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:59.081+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:59.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:05:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:05:59.081+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:05:59.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:58.919+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:59.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:58.919+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:59.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:58.919+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:05:59.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:59.627+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:59.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:59.627+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:05:59.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:05:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:05:59.627+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:00.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:59.951+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:00.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:59.951+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:00.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:05:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:05:59.951+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:00.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:00.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:00.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:00.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:00.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:00.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:00.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:00.638+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:00.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:00.638+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:00.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:00.638+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:01.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:00.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:01.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:00.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:01.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:00.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:01.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:00.989+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:01.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:00.989+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:01.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:00.989+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:01.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:01.667+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:01.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:01.667+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:01.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:01.667+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:02.393 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:01.989+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:02.393 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:01.989+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:02.393 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:01.989+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:02.393 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:02.014+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:02.393 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:02.014+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:02.394 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:02.014+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:02.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:02.645+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:02.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:02.645+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:02.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:02.645+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:03.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:02.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:03.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:02.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:03.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:02.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:03.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:02.988+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:03.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:02.988+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:03.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:02.988+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:03.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.463+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:06:03.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.463+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:06:03.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.463+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcff001c990 tx=0x7fcff0028000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:06:03.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.463+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcff001c990 tx=0x7fcff0028000 comp rx=0 tx=0).stop 2024-04-27T04:06:03.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.463+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:06:03.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.463+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:03.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.464+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:03.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.464+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:03.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcfe40412a0 2024-04-27T04:06:03.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044fd0 con 0x7fcff8101eb0 2024-04-27T04:06:03.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.464+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:06:03.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.465+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:06:03.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.465+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:03.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.465+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40412a0 2024-04-27T04:06:03.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.465+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80317d0 tx=0x7fcff00462d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:06:03.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.465+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8013a90 con 0x7fcfe40412a0 2024-04-27T04:06:03.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.466+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe800fe50 con 0x7fcfe40412a0 2024-04-27T04:06:03.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:03.467+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013c10 con 0x7fcfe40412a0 2024-04-27T04:06:03.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:06:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:06:03.463+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:06:03.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:03.618+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:03.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:03.618+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:03.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:03.618+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:04.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:03.957+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:04.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:03.957+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:04.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:03.957+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:04.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:04.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:04.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:04.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:04.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:04.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:04.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:04.658+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:04.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:04.658+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:04.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:04.658+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:04.996+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:04.996+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:04.996+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:05.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:04.991+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:05.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:04.991+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:05.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:04.991+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:05.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:05.650+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:05.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:05.650+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:05.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:05.650+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:06.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:05.969+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:06.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:05.969+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:06.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:05.969+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:06.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:06.009+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:06.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:06.009+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:06.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:06.009+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:06.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:06.605+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:06.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:06.605+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:06.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:06.605+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:07.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:06.968+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:07.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:06.968+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:07.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:06.968+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:07.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:06.988+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:07.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:06.988+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:07.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:06.988+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:07.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:07.583+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:07.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:07.583+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:07.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:07.583+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:08.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:07.943+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:08.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:07.943+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:08.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:07.943+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:08.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:08.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:08.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:08.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:08.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:08.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:08.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.464+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:06:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.464+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:06:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.464+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80317d0 tx=0x7fcff00462d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:06:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.464+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80317d0 tx=0x7fcff00462d0 comp rx=0 tx=0).stop 2024-04-27T04:06:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:06:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.464+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.464+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.464+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcfe40443d0 2024-04-27T04:06:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044fd0 con 0x7fcff8101eb0 2024-04-27T04:06:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.464+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:06:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.465+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:06:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.465+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.465+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40443d0 2024-04-27T04:06:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.466+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80301a0 tx=0x7fcfe804fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:06:08.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.467+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800fe50 con 0x7fcfe40443d0 2024-04-27T04:06:08.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.467+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe800fe50 con 0x7fcfe40443d0 2024-04-27T04:06:08.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:08.469+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe803a630 con 0x7fcfe40443d0 2024-04-27T04:06:08.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:06:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:06:08.464+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:06:08.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:08.604+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:08.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:08.604+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:08.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:08.604+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:09.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:08.946+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:09.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:08.946+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:09.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:08.946+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:09.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:09.006+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:09.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:09.006+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:09.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:09.006+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:09.904 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:09.569+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:09.904 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:09.569+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:09.904 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:09.569+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:10.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:09.904+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:10.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:09.904+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:10.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:09.904+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:10.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:09.962+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:10.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:09.962+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:10.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:09.962+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:10.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:10.582+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:06:10.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:10.582+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:06:10.893 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:10.583+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:10.894 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:10.583+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:10.894 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:10.583+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:11.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:10.894+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:11.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:10.894+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:11.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:10.894+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:11.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:10.929+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:11.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:10.929+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:11.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:10.929+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:11.844 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:11.607+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:11.845 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:11.607+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:11.845 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:11.607+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:12.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:11.948+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:12.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:11.948+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:12.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:11.948+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:12.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:11.844+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:12.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:11.844+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:12.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:11.844+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:12.809 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:12.652+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:12.809 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:12.652+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:12.809 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:12.652+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:13.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:12.964+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:13.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:12.964+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:13.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:12.964+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:13.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:12.809+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:13.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:12.809+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:13.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:12.809+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:13.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.464+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:06:13.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.464+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:06:13.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.464+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80301a0 tx=0x7fcfe804fa10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:06:13.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.464+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80301a0 tx=0x7fcfe804fa10 comp rx=0 tx=0).stop 2024-04-27T04:06:13.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:06:13.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.464+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:13.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.464+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.464+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051040 con 0x7fcfe40412a0 2024-04-27T04:06:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcff8101eb0 2024-04-27T04:06:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.464+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:06:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.465+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:06:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.465+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.465+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40412a0 2024-04-27T04:06:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.465+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fcff0008000 tx=0x7fcfe8014900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:06:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.466+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0030050 con 0x7fcfe40412a0 2024-04-27T04:06:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.466+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0001e30 con 0x7fcfe40412a0 2024-04-27T04:06:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:13.467+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0011a20 con 0x7fcfe40412a0 2024-04-27T04:06:13.764 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:06:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:06:13.464+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 28 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:06:13.765 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:13.700+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:13.765 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:13.700+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:13.765 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:13.700+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:14.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:13.969+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:14.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:13.969+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:14.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:13.969+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:14.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:13.765+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:14.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:13.765+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:14.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:13.765+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:15.002 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:14.765+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:15.002 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:14.765+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:15.002 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:14.765+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:15.002 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:14.731+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:15.002 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:14.731+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:15.002 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:14.731+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:15.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:15.002+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:15.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:15.002+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:15.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:15.002+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:16.035 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:15.699+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:16.035 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:15.699+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:16.035 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:15.699+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:16.035 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:15.751+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:16.035 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:15.751+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:16.035 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:15.751+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:16.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:16.035+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:16.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:16.035+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:16.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:16.035+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:16.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:16.667+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:16.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:16.667+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:16.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:16.667+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:16.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:16.772+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:16.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:16.772+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:16.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:16.772+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:17.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:17.060+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:17.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:17.061+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:17.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:17.061+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:17.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:17.740+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:17.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:17.740+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:17.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:17.740+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:17.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:17.623+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:17.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:17.623+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:17.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:17.623+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:18.083+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:18.083+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:18.083+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:18.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.464+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:06:18.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.464+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:06:18.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.464+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fcff0008000 tx=0x7fcfe8014900 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:06:18.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.464+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fcff0008000 tx=0x7fcfe8014900 comp rx=0 tx=0).stop 2024-04-27T04:06:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:06:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.464+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.464+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.464+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4050e10 con 0x7fcfe40443d0 2024-04-27T04:06:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e770 con 0x7fcff8101eb0 2024-04-27T04:06:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.465+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:06:18.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.465+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:06:18.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.465+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:18.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.465+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044e90 con 0x7fcfe40443d0 2024-04-27T04:06:18.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.466+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8032f70 tx=0x7fcff0041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:06:18.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.466+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe80465e0 con 0x7fcfe40443d0 2024-04-27T04:06:18.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.466+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe800f750 con 0x7fcfe40443d0 2024-04-27T04:06:18.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:18.467+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013a90 con 0x7fcfe40443d0 2024-04-27T04:06:18.727 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:06:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:06:18.465+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 50 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:06:18.728 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:18.670+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:18.728 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:18.670+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:18.728 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:18.670+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:19.131 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:18.728+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:19.131 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:18.728+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:19.131 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:18.728+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:19.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:19.131+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:19.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:19.131+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:19.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:19.131+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:20.125 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:19.757+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:20.125 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:19.757+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:20.125 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:19.757+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:20.125 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:19.718+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:20.125 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:19.718+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:20.125 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:19.718+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:20.125+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:20.125+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:20.125+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:21.134 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:20.756+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:21.134 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:20.756+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:21.134 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:20.756+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:21.135 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:20.800+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:21.135 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:20.800+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:21.135 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:20.800+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:21.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:21.133+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:21.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:21.133+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:21.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:21.133+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:21.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:21.762+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:21.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:21.762+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:21.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:21.762+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:21.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:21.786+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:21.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:21.786+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:21.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:21.786+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:22.109+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:22.109+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:22.109+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:22.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:22.773+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:22.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:22.773+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:22.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:22.773+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:22.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:22.743+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:22.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:22.743+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:22.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:22.743+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:23.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:23.123+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:23.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:23.123+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:23.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:23.123+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:23.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.464+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:06:23.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.464+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:06:23.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.464+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8032f70 tx=0x7fcff0041040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:06:23.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.464+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8032f70 tx=0x7fcff0041040 comp rx=0 tx=0).stop 2024-04-27T04:06:23.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:06:23.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.464+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:23.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.464+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:23.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.464+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:23.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e770 con 0x7fcfe40412a0 2024-04-27T04:06:23.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.464+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcff8101eb0 2024-04-27T04:06:23.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.465+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:06:23.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.465+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:06:23.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.465+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:23.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.465+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40412a0 2024-04-27T04:06:23.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.466+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fcff0032c10 tx=0x7fcfe803a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:06:23.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.466+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0030040 con 0x7fcfe40412a0 2024-04-27T04:06:23.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.467+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff00340c0 con 0x7fcfe40412a0 2024-04-27T04:06:23.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:23.468+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0001ce0 con 0x7fcfe40412a0 2024-04-27T04:06:23.733 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:06:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:06:23.464+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 66 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:06:24.101 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:23.733+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:24.101 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:23.733+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:24.101 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:23.733+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:24.102 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:23.768+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:24.102 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:23.768+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:24.102 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:23.768+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:24.100+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:24.100+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:24.100+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:25.075 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:24.702+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:25.075 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:24.702+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:25.075 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:24.702+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:25.075 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:24.783+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:25.076 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:24.783+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:25.076 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:24.783+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:25.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:25.074+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:25.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:25.074+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:25.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:25.074+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:25.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:25.940+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:06:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:25.940+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:06:26.073 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:25.749+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:26.073 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:25.749+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:26.073 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:25.749+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:26.074 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:25.742+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:26.074 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:25.742+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:26.074 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:25.742+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:26.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:26.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:26.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:26.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:26.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:26.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:27.114 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:26.734+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:27.115 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:26.734+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:27.115 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:26.734+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:27.115 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:26.777+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:27.115 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:26.777+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:27.115 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:26.777+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:27.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:27.114+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:27.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:27.114+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:27.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:27.114+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:28.117 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:27.707+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:28.117 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:27.707+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:28.117 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:27.707+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:28.117 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:27.750+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:28.117 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:27.750+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:28.117 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:27.750+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:28.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:28.116+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:28.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:28.116+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:28.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:28.116+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:28.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.464+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:06:28.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.464+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:06:28.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.465+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fcff0032c10 tx=0x7fcfe803a0b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:06:28.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.465+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fcff0032c10 tx=0x7fcfe803a0b0 comp rx=0 tx=0).stop 2024-04-27T04:06:28.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.465+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:06:28.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.465+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:28.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.465+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:28.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.465+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:28.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.465+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e6c0 con 0x7fcfe40443d0 2024-04-27T04:06:28.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.465+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcff8101eb0 2024-04-27T04:06:28.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.465+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:06:28.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.466+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:06:28.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.466+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:28.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.466+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40443d0 2024-04-27T04:06:28.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.466+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8039e10 tx=0x7fcff0009e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:06:28.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.466+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800fe50 con 0x7fcfe40443d0 2024-04-27T04:06:28.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.466+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe801b8c0 con 0x7fcfe40443d0 2024-04-27T04:06:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:28.467+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe803a630 con 0x7fcfe40443d0 2024-04-27T04:06:28.780 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:06:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:06:28.464+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 83 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:06:28.780 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:28.688+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:28.781 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:28.688+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:28.781 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:28.688+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:29.116 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:28.779+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:29.117 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:28.779+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:29.117 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:28.779+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:29.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:29.115+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:29.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:29.115+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:29.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:29.115+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:30.099 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:29.801+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:30.099 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:29.801+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:30.099 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:29.801+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:30.100 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:29.717+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:30.100 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:29.717+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:30.100 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:29.717+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:30.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:30.098+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:30.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:30.098+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:30.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:30.098+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:31.146 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:30.676+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:31.146 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:30.676+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:31.146 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:30.676+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:31.146 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:30.764+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:31.146 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:30.764+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:31.146 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:30.764+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:31.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:31.146+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:31.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:31.146+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:31.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:31.146+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:31.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:31.745+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:31.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:31.745+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:31.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:31.745+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:31.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:31.657+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:31.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:31.657+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:31.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:31.657+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:32.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:32.170+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:32.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:32.170+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:32.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:32.170+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:32.742 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:32.653+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:32.742 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:32.653+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:32.742 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:32.653+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:33.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:32.787+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:33.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:32.787+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:33.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:32.787+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.465+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:06:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.465+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:06:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.465+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8039e10 tx=0x7fcff0009e80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:06:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.465+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8039e10 tx=0x7fcff0009e80 comp rx=0 tx=0).stop 2024-04-27T04:06:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.465+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:06:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.465+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.465+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.465+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.465+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e6c0 con 0x7fcfe40412a0 2024-04-27T04:06:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.465+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcff8101eb0 2024-04-27T04:06:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.466+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:06:33.467 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:33.215+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:33.467 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:33.215+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:33.467 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:33.215+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.466+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:06:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.466+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:33.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.466+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40412a0 2024-04-27T04:06:33.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.467+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fcff0030f60 tx=0x7fcfe804fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:06:33.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.467+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0011a20 con 0x7fcfe40412a0 2024-04-27T04:06:33.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.467+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0011a20 con 0x7fcfe40412a0 2024-04-27T04:06:33.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:33.468+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0032900 con 0x7fcfe40412a0 2024-04-27T04:06:33.788 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:06:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:06:33.465+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 101 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:06:33.789 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:33.631+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:33.789 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:33.631+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:33.789 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:33.631+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:34.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:33.788+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:34.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:33.788+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:34.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:33.788+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:34.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:34.195+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:34.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:34.195+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:34.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:34.195+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:34.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:34.671+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:34.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:34.671+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:34.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:34.671+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:34.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:34.755+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:34.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:34.755+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:34.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:34.755+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:35.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:35.191+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:35.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:35.191+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:35.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:35.191+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:35.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:35.658+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:35.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:35.658+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:35.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:35.658+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:35.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:35.779+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:35.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:35.779+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:35.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:35.779+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:36.660 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:36.210+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:36.661 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:36.210+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:36.661 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:36.210+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:36.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:36.661+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:36.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:36.661+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:36.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:36.661+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:36.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:36.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:36.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:36.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:36.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:36.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:37.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:37.249+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:37.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:37.249+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:37.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:37.249+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:38.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:37.691+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:38.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:37.691+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:38.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:37.691+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:38.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:37.785+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:38.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:37.785+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:38.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:37.785+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:38.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.466+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:06:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.466+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:06:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.466+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fcff0030f60 tx=0x7fcfe804fa10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:06:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.466+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fcff0030f60 tx=0x7fcfe804fa10 comp rx=0 tx=0).stop 2024-04-27T04:06:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.467+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:06:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.467+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.467+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.467+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.467+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e6c0 con 0x7fcfe40443d0 2024-04-27T04:06:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.467+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcff8101eb0 2024-04-27T04:06:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.467+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:06:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.468+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:06:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.468+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.468+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40443d0 2024-04-27T04:06:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.468+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fcff003ee50 tx=0x7fcff0041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:06:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.469+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe40443d0 2024-04-27T04:06:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.469+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff002b7c0 con 0x7fcfe40443d0 2024-04-27T04:06:38.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:38.470+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff001f120 con 0x7fcfe40443d0 2024-04-27T04:06:38.659 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:06:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:06:38.467+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 119 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:06:38.660 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:38.264+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:38.660 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:38.264+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:38.660 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:38.264+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:38.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:38.660+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:38.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:38.660+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:38.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:38.660+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:38.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:38.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:38.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:38.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:38.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:38.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:39.629 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:39.222+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:39.629 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:39.222+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:39.629 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:39.222+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:39.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:39.850+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:39.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:39.850+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:39.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:39.850+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:39.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:39.630+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:39.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:39.630+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:39.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:39.630+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:40.662 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:40.228+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:40.662 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:40.228+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:40.662 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:40.228+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:40.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:40.844+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:40.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:40.844+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:40.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:40.844+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:40.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:40.662+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:40.926 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:40.662+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:40.926 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:40.662+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:41.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:41.301+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:06:41.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:41.301+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:06:41.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:41.271+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:41.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:41.271+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:41.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:41.271+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:41.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:41.625+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:41.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:41.625+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:41.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:41.625+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:41.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:41.815+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:41.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:41.815+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:41.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:41.815+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:42.636 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:42.259+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:42.637 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:42.259+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:42.637 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:42.259+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:42.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:42.856+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:42.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:42.856+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:42.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:42.856+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:42.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:42.637+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:42.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:42.637+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:42.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:42.637+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:43.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.467+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:06:43.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.467+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:06:43.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.467+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fcff003ee50 tx=0x7fcff0041040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:06:43.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.467+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fcff003ee50 tx=0x7fcff0041040 comp rx=0 tx=0).stop 2024-04-27T04:06:43.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.467+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:06:43.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.467+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:43.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.467+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:43.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.467+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:43.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.467+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e6c0 con 0x7fcfe40412a0 2024-04-27T04:06:43.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.467+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcff8101eb0 2024-04-27T04:06:43.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.467+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:06:43.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.468+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:06:43.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.468+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:43.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.468+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40412a0 2024-04-27T04:06:43.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.468+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fcff00495c0 tx=0x7fcff0042f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:06:43.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.469+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0030070 con 0x7fcfe40412a0 2024-04-27T04:06:43.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.469+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff002b7c0 con 0x7fcfe40412a0 2024-04-27T04:06:43.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:43.470+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff00462d0 con 0x7fcfe40412a0 2024-04-27T04:06:43.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:06:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:06:43.467+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 136 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:06:43.516 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:43.264+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:43.516 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:43.264+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:43.516 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:43.264+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:43.879 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:43.603+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:43.880 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:43.603+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:43.880 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:43.603+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:44.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:43.880+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:44.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:43.880+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:44.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:43.880+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:44.586 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:44.234+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:44.586 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:44.234+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:44.586 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:44.234+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:44.876 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:44.586+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:44.877 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:44.586+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:44.877 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:44.586+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:45.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:44.877+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:45.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:44.877+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:45.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:44.877+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:45.615 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:45.266+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:45.615 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:45.266+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:45.615 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:45.266+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:45.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:45.615+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:45.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:45.615+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:45.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:45.615+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:46.229 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:45.925+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:46.229 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:45.925+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:46.229 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:45.925+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:46.651 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:46.230+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:46.651 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:46.230+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:46.651 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:46.230+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:46.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:46.651+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:46.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:46.651+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:46.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:46.651+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:46.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:46.880+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:46.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:46.880+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:46.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:46.880+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:47.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:47.265+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:47.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:47.265+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:47.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:47.265+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:48.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:47.690+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:48.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:47.690+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:48.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:47.690+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:48.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:47.849+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:48.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:47.849+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:48.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:47.849+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:48.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.467+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:06:48.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.467+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:06:48.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.467+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fcff00495c0 tx=0x7fcff0042f40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:06:48.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.468+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fcff00495c0 tx=0x7fcff0042f40 comp rx=0 tx=0).stop 2024-04-27T04:06:48.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.468+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:06:48.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.468+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:48.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.468+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:48.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.468+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:48.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.468+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e6c0 con 0x7fcfe40443d0 2024-04-27T04:06:48.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.468+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcff8101eb0 2024-04-27T04:06:48.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.468+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:06:48.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.469+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:06:48.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.469+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:48.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.469+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40443d0 2024-04-27T04:06:48.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.470+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8045510 tx=0x7fcff0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:06:48.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.470+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800cea0 con 0x7fcfe40443d0 2024-04-27T04:06:48.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.471+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe803c1b0 con 0x7fcfe40443d0 2024-04-27T04:06:48.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:48.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe803a630 con 0x7fcfe40443d0 2024-04-27T04:06:48.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:06:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:06:48.467+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 153 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:06:48.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:48.271+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:48.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:48.271+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:48.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:48.271+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:49.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:48.693+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:49.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:48.693+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:49.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:48.693+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:49.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:48.830+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:49.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:48.830+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:49.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:48.830+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:49.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:49.271+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:49.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:49.271+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:49.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:49.271+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:50.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:49.698+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:50.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:49.698+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:50.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:49.698+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:50.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:49.848+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:50.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:49.848+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:50.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:49.848+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:50.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:50.269+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:50.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:50.269+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:50.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:50.269+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:51.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:50.827+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:51.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:50.827+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:51.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:50.827+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:51.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:50.741+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:51.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:50.741+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:51.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:50.741+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:51.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:51.280+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:51.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:51.280+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:51.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:51.280+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:51.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:51.794+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:51.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:51.794+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:51.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:51.794+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:51.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:51.727+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:51.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:51.727+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:51.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:51.727+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:52.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:52.298+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:52.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:52.298+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:52.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:52.298+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:53.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:52.785+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:53.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:52.785+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:53.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:52.785+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:53.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:52.715+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:53.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:52.715+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:53.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:52.715+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.468+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:06:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.468+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:06:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.468+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8045510 tx=0x7fcff0000bc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:06:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.468+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8045510 tx=0x7fcff0000bc0 comp rx=0 tx=0).stop 2024-04-27T04:06:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.468+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:06:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.468+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.468+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.468+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.468+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e770 con 0x7fcfe40412a0 2024-04-27T04:06:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.468+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcff8101eb0 2024-04-27T04:06:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.468+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:06:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.469+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:06:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.469+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.469+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40412a0 2024-04-27T04:06:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.470+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fcfe802b610 tx=0x7fcfe803a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:06:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.470+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800cea0 con 0x7fcfe40412a0 2024-04-27T04:06:53.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.471+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8043b60 con 0x7fcfe40412a0 2024-04-27T04:06:53.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:53.472+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013a90 con 0x7fcfe40412a0 2024-04-27T04:06:53.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:06:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:06:53.468+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 170 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:06:53.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:53.342+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:53.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:53.342+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:53.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:53.342+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:54.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:53.791+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:54.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:53.791+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:54.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:53.791+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:54.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:53.755+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:54.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:53.755+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:54.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:53.755+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:54.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:54.299+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:54.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:54.299+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:54.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:54.299+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:54.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:54.301+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:06:55.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:54.718+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:55.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:54.718+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:55.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:54.718+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:55.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:54.720+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:06:55.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:54.821+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:55.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:54.821+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:55.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:54.821+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:55.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:54.823+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:06:55.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:55.265+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:55.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:55.265+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:55.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:55.265+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:55.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:55.266+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:06:56.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:55.738+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:56.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:55.738+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:56.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:55.738+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:56.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:55.740+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:06:56.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:55.833+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:56.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:55.833+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:56.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:55.833+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:56.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:55.834+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:06:56.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:56.661+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:06:56.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:56.661+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:06:56.661 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:56.224+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:56.661 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:56.224+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:56.661 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:56.224+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:56.661 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:56.225+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:06:56.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:56.720+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:56.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:56.720+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:56.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:56.720+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:56.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:56.721+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:06:56.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:56.857+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:56.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:56.857+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:56.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:56.857+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:56.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:56.857+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:06:57.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:57.193+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:57.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:57.193+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:57.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:57.193+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:57.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:57.194+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:06:57.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:57.743+0000 7fcfee7fc700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfdc003550 con 0x7fcfe40412a0 2024-04-27T04:06:58.008 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:57.738+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:58.008 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:57.738+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:58.008 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:57.738+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:58.008 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:57.739+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:06:58.008 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:57.837+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:58.008 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:57.837+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:58.008 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:57.837+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:58.008 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:57.838+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:06:58.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:58.167+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:58.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:58.167+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:58.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:58.167+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:58.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:58.168+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:06:58.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.468+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:06:58.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.468+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:06:58.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.468+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fcfe802b610 tx=0x7fcfe803a040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:06:58.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.468+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fcfe802b610 tx=0x7fcfe803a040 comp rx=0 tx=0).stop 2024-04-27T04:06:58.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.469+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:06:58.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.469+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:58.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.469+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:58.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.469+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:06:58.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.469+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e790 con 0x7fcfe40443d0 2024-04-27T04:06:58.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.469+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcff8101eb0 2024-04-27T04:06:58.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.469+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:06:58.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.469+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:06:58.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.469+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:06:58.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.469+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40443d0 2024-04-27T04:06:58.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.470+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8037ff0 tx=0x7fcfe8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:06:58.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.470+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8007b80 con 0x7fcfe40443d0 2024-04-27T04:06:58.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.470+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe80079a0 con 0x7fcfe40443d0 2024-04-27T04:06:58.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:06:58.471+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013a90 con 0x7fcfe40443d0 2024-04-27T04:06:58.749 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:06:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:06:58.468+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 188 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:06:59.159 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:58.749+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:59.159 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:58.749+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:59.159 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:58.749+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:06:59.159 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:58.750+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:06:59.159 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:58.856+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:59.160 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:58.856+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:59.160 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:58.856+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:06:59.160 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:58.857+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:06:59.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:59.158+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:59.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:59.158+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:59.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:59.158+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:06:59.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:06:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:06:59.160+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:00.152 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:59.850+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:00.152 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:59.850+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:00.152 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:59.850+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:00.152 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:06:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:06:59.850+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:00.153 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:59.753+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:00.153 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:59.753+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:00.153 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:59.753+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:00.153 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:06:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:06:59.754+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:00.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:00.152+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:00.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:00.152+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:00.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:00.152+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:00.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:00.154+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:01.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:00.720+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:01.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:00.720+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:01.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:00.720+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:01.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:00.720+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:01.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:00.808+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:01.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:00.808+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:01.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:00.808+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:01.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:00.809+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:01.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:01.190+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:01.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:01.190+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:01.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:01.190+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:01.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:01.191+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:01.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:01.715+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:01.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:01.715+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:01.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:01.715+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:01.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:01.716+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:01.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:01.805+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:01.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:01.805+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:01.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:01.805+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:01.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:01.805+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:02.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:02.228+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:02.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:02.228+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:02.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:02.228+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:02.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:02.229+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:03.063 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:02.680+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:03.063 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:02.680+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:03.064 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:02.680+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:03.064 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:02.680+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:03.064 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:02.775+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:03.064 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:02.775+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:03.064 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:02.775+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:03.064 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:02.776+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:03.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:03.219+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:03.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:03.219+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:03.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:03.219+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:03.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:03.220+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:03.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.468+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:07:03.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.469+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:07:03.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.469+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8037ff0 tx=0x7fcfe8009f90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:07:03.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.469+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8037ff0 tx=0x7fcfe8009f90 comp rx=0 tx=0).stop 2024-04-27T04:07:03.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.469+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:07:03.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.469+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:03.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.469+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:03.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.469+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:03.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.469+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e770 con 0x7fcfe40412a0 2024-04-27T04:07:03.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.469+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcff8101eb0 2024-04-27T04:07:03.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.469+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:07:03.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.470+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:07:03.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.470+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:03.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.470+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40412a0 2024-04-27T04:07:03.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.470+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcff001af30 tx=0x7fcfe8014d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:07:03.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.471+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0030040 con 0x7fcfe40412a0 2024-04-27T04:07:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.474+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff000ce90 con 0x7fcfe40412a0 2024-04-27T04:07:03.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:03.475+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe40412a0 2024-04-27T04:07:03.803 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:07:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:07:03.469+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 204 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:07:03.803 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:03.666+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:03.803 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:03.666+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:03.803 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:03.666+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:03.803 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:03.666+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:04.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:03.803+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:04.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:03.803+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:04.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:03.803+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:04.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:03.804+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:04.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:04.248+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:04.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:04.248+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:04.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:04.248+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:04.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:04.248+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:05.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:04.758+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:05.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:04.758+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:05.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:04.759+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:05.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:04.760+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:05.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:04.712+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:05.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:04.712+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:05.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:04.712+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:05.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:04.712+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:05.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:05.244+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:05.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:05.244+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:05.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:05.244+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:05.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:05.245+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:05.745+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:05.745+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:05.745+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:05.746+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:06.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:05.704+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:06.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:05.704+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:06.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:05.704+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:06.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:05.705+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:06.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:06.255+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:06.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:06.255+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:06.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:06.255+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:06.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:06.256+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:07.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:06.735+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:07.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:06.735+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:07.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:06.735+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:07.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:06.736+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:07.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:06.748+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:07.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:06.748+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:07.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:06.748+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:07.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:06.748+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:07.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:07.216+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:07.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:07.216+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:07.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:07.216+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:07.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:07.217+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:08.125 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:07.739+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:08.125 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:07.739+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:08.125 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:07.739+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:08.125 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:07.740+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:08.125 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:07.750+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:08.125 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:07.750+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:08.125 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:07.750+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:08.125 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:07.750+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:08.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:08.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:08.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:08.183+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:08.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.469+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:07:08.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.469+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:07:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.469+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcff001af30 tx=0x7fcfe8014d20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:07:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.469+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcff001af30 tx=0x7fcfe8014d20 comp rx=0 tx=0).stop 2024-04-27T04:07:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.470+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:07:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.470+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.470+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.470+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.470+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4043a40 con 0x7fcfe40443d0 2024-04-27T04:07:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.470+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e770 con 0x7fcff8101eb0 2024-04-27T04:07:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.470+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:07:08.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.471+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:07:08.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.471+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:08.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.471+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044e90 con 0x7fcfe40443d0 2024-04-27T04:07:08.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.471+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400a670 tx=0x7fcff0006a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:07:08.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0030040 con 0x7fcfe40443d0 2024-04-27T04:07:08.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe40443d0 2024-04-27T04:07:08.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:08.474+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff002b5c0 con 0x7fcfe40443d0 2024-04-27T04:07:08.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:07:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:07:08.469+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 221 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:07:08.743 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:08.701+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:08.743 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:08.701+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:08.743 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:08.701+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:08.743 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:08.703+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:09.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:08.743+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:09.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:08.743+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:09.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:08.743+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:09.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:08.744+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:09.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:09.233+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:09.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:09.233+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:09.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:09.233+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:09.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:09.235+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:10.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:09.727+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:10.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:09.727+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:10.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:09.727+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:10.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:09.728+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:10.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:09.764+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:10.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:09.764+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:10.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:09.764+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:10.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:09.765+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:10.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:10.199+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:10.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:10.199+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:10.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:10.199+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:10.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:10.199+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:11.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:10.737+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:11.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:10.737+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:11.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:10.737+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:11.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:10.737+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:11.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:10.797+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:11.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:10.797+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:11.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:10.797+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:11.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:10.797+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:11.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:11.175+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:11.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:11.175+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:11.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:11.175+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:11.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:11.176+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:11.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:11.747+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:11.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:11.747+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:11.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:11.747+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:11.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:11.748+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:11.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:11.819+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:11.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:11.819+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:11.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:11.819+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:11.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:11.820+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:12.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:12.021+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:07:12.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:12.021+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:07:12.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:12.220+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:12.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:12.220+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:12.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:12.220+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:12.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:12.220+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:13.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:12.793+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:13.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:12.793+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:13.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:12.793+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:13.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:12.794+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:13.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:12.800+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:13.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:12.800+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:13.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:12.800+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:13.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:12.801+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:13.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.470+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:07:13.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.470+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:07:13.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.470+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400a670 tx=0x7fcff0006a60 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:07:13.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.470+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400a670 tx=0x7fcff0006a60 comp rx=0 tx=0).stop 2024-04-27T04:07:13.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.470+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:07:13.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.470+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:13.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.470+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:13.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.470+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:13.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.470+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e6c0 con 0x7fcfe40412a0 2024-04-27T04:07:13.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.470+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcff8101eb0 2024-04-27T04:07:13.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.470+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:07:13.471 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:13.196+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:13.471 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:13.196+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:13.471 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:13.196+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:13.471 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:13.196+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:13.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.470+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:07:13.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.470+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:13.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.470+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40412a0 2024-04-27T04:07:13.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.471+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40471b0 tx=0x7fcff001f120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:07:13.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.471+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0030070 con 0x7fcfe40412a0 2024-04-27T04:07:13.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.472+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0017270 con 0x7fcfe40412a0 2024-04-27T04:07:13.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:13.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0010570 con 0x7fcfe40412a0 2024-04-27T04:07:13.818 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:07:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:07:13.470+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 238 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:07:14.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:13.819+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:14.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:13.819+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:14.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:13.819+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:14.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:13.820+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:13.822+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:13.822+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:14.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:13.822+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:14.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:13.823+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:14.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:14.185+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:14.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:14.185+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:14.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:14.185+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:14.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:14.187+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:15.092 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:14.804+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:15.093 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:14.804+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:15.093 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:14.804+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:15.093 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:14.805+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:15.093 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:14.805+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:15.093 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:14.805+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:15.093 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:14.805+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:15.093 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:14.806+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:15.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:15.231+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:15.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:15.231+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:15.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:15.231+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:15.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:15.232+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:16.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:15.845+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:16.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:15.845+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:16.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:15.845+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:16.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:15.846+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:16.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:15.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:16.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:15.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:16.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:15.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:16.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:15.806+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:16.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:16.195+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:16.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:16.195+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:16.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:16.195+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:16.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:16.196+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:17.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:16.894+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:17.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:16.894+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:17.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:16.894+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:17.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:16.896+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:17.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:16.814+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:17.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:16.814+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:17.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:16.814+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:17.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:16.815+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:17.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:17.193+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:17.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:17.193+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:17.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:17.193+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:17.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:17.194+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:18.157 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:17.895+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:18.157 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:17.895+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:18.157 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:17.895+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:18.157 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:17.895+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:18.158 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:17.847+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:18.158 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:17.847+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:18.158 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:17.847+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:18.158 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:17.848+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:18.157+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:18.157+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:18.157+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:18.158+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:18.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.470+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:07:18.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.470+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:07:18.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.470+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40471b0 tx=0x7fcff001f120 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:07:18.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.470+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40471b0 tx=0x7fcff001f120 comp rx=0 tx=0).stop 2024-04-27T04:07:18.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.470+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:07:18.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.470+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:18.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.470+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:18.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.470+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:18.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.470+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e6c0 con 0x7fcfe40443d0 2024-04-27T04:07:18.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.470+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcff8101eb0 2024-04-27T04:07:18.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.471+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:07:18.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.471+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:07:18.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.471+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:18.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.471+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40443d0 2024-04-27T04:07:18.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.472+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400fe10 tx=0x7fcff0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:07:18.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.472+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00103a0 con 0x7fcfe40443d0 2024-04-27T04:07:18.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.472+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff002b790 con 0x7fcfe40443d0 2024-04-27T04:07:18.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:18.475+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0010cd0 con 0x7fcfe40443d0 2024-04-27T04:07:18.867 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:07:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:07:18.470+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 256 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:07:19.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:18.881+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:19.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:18.881+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:19.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:18.881+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:19.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:18.881+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:19.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:18.868+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:19.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:18.868+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:19.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:18.868+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:19.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:18.868+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:19.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:19.195+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:19.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:19.195+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:19.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:19.195+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:19.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:19.196+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:20.148 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:19.859+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:20.148 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:19.859+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:20.149 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:19.859+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:20.149 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:19.860+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:20.149 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:19.892+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:20.149 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:19.892+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:20.149 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:19.892+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:20.149 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:19.892+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:20.148+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:20.148+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:20.148+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:20.149+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:21.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:20.819+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:21.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:20.819+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:21.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:20.819+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:21.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:20.820+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:21.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:20.851+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:21.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:20.851+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:21.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:20.851+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:21.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:20.851+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:21.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:21.177+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:21.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:21.177+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:21.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:21.177+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:21.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:21.177+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:22.140 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:21.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:22.140 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:21.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:22.140 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:21.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:22.140 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:21.822+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:22.141 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:21.843+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:22.141 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:21.843+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:22.141 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:21.843+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:22.141 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:21.844+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:22.140+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:22.140+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:22.140+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:22.141+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:23.136 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:22.871+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:23.136 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:22.871+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:23.136 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:22.871+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:23.136 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:22.872+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:23.137 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:22.858+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:23.137 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:22.858+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:23.137 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:22.858+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:23.137 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:22.859+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:23.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:23.136+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:23.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:23.136+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:23.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:23.136+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:23.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:23.138+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:23.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.471+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:07:23.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.471+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:07:23.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.471+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400fe10 tx=0x7fcff0000bc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:07:23.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.471+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400fe10 tx=0x7fcff0000bc0 comp rx=0 tx=0).stop 2024-04-27T04:07:23.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.471+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:07:23.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.471+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:23.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.471+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:23.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.471+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:23.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.471+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e6c0 con 0x7fcfe40412a0 2024-04-27T04:07:23.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.471+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcff8101eb0 2024-04-27T04:07:23.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.471+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:07:23.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.472+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:07:23.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.472+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:23.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.472+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40412a0 2024-04-27T04:07:23.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.472+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcfe401aea0 tx=0x7fcff0011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:07:23.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800cea0 con 0x7fcfe40412a0 2024-04-27T04:07:23.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe803a810 con 0x7fcfe40412a0 2024-04-27T04:07:23.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:23.474+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe804b1f0 con 0x7fcfe40412a0 2024-04-27T04:07:23.860 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:07:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:07:23.471+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 272 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:07:24.124 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:23.901+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:24.124 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:23.901+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:24.124 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:23.901+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:24.124 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:23.902+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:24.124 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:23.860+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:24.124 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:23.860+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:24.124 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:23.860+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:24.124 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:23.861+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:24.124+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:24.124+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:24.124+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:24.125+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:25.168 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:24.871+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:25.169 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:24.871+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:25.169 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:24.871+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:25.169 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:24.871+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:25.169 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:24.902+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:25.169 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:24.902+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:25.169 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:24.902+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:25.169 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:24.903+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:25.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:25.169+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:25.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:25.169+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:25.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:25.169+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:25.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:25.169+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:26.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:25.903+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:26.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:25.903+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:26.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:25.903+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:26.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:25.904+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:26.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:25.931+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:26.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:25.931+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:26.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:25.931+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:26.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:25.932+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:26.219+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:26.219+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:26.219+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:26.220+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:27.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:26.918+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:27.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:26.918+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:27.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:26.918+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:27.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:26.919+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:27.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:26.959+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:27.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:26.959+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:27.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:26.959+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:27.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:26.960+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:27.248 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:27.247+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:27.248 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:27.247+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:27.248 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:27.247+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:27.248 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:27.247+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:27.381+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:07:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:27.381+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:07:28.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:27.910+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:28.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:27.910+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:28.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:27.910+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:28.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:27.911+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:28.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:27.975+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:28.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:27.975+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:28.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:27.975+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:28.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:27.976+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:28.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.471+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:07:28.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.471+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:07:28.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.471+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcfe401aea0 tx=0x7fcff0011a20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:07:28.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.471+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcfe401aea0 tx=0x7fcff0011a20 comp rx=0 tx=0).stop 2024-04-27T04:07:28.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.471+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:07:28.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.471+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:28.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.471+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:28.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.471+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:28.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.471+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e6c0 con 0x7fcfe40443d0 2024-04-27T04:07:28.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.471+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcff8101eb0 2024-04-27T04:07:28.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.472+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:07:28.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.473+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:07:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.473+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.473+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40443d0 2024-04-27T04:07:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.473+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800b980 tx=0x7fcfe803a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:07:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800cea0 con 0x7fcfe40443d0 2024-04-27T04:07:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.474+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe801ba90 con 0x7fcfe40443d0 2024-04-27T04:07:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:28.474+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013a90 con 0x7fcfe40443d0 2024-04-27T04:07:28.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:07:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:07:28.471+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 289 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:07:28.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:28.270+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:28.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:28.270+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:28.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:28.270+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:28.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:28.271+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:29.311 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:28.961+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:29.311 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:28.961+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:29.312 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:28.961+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:29.312 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:28.962+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:29.312 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:29.018+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:29.312 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:29.018+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:29.312 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:29.018+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:29.312 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:29.019+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:29.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:29.311+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:29.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:29.311+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:29.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:29.311+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:29.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:29.313+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:30.328 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:29.983+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:30.328 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:29.983+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:30.328 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:29.983+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:30.328 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:29.984+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:30.328 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:30.000+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:30.329 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:30.000+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:30.329 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:30.000+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:30.329 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:30.001+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:30.328+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:30.328+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:30.328+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:30.329+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:31.329 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:31.007+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:31.330 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:31.007+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:31.330 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:31.007+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:31.330 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:31.007+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:31.330 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:31.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:31.330 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:31.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:31.330 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:31.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:31.330 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:31.003+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:31.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:31.330+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:31.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:31.330+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:31.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:31.330+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:31.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:31.331+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:32.343 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:32.048+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:32.343 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:32.048+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:32.343 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:32.048+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:32.343 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:32.049+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:32.343 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:32.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:32.343 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:32.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:32.343 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:32.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:32.344 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:32.030+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:32.343+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:32.343+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:32.343+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:32.344+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:33.312 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:33.013+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:33.312 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:33.013+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:33.313 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:33.013+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:33.313 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:33.013+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:33.313 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:33.042+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:33.313 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:33.042+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:33.313 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:33.042+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:33.313 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:33.043+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.471+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:07:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.471+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:07:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.471+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800b980 tx=0x7fcfe803a040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:07:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.471+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800b980 tx=0x7fcfe803a040 comp rx=0 tx=0).stop 2024-04-27T04:07:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.472+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:07:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.472+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.472+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.472+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.472+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e6c0 con 0x7fcfe40412a0 2024-04-27T04:07:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.472+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcff8101eb0 2024-04-27T04:07:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.472+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:07:33.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.472+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:07:33.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.472+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:33.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.473+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40452a0 con 0x7fcfe40412a0 2024-04-27T04:07:33.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.473+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8014940 tx=0x7fcfe8024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:07:33.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.474+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800cea0 con 0x7fcfe40412a0 2024-04-27T04:07:33.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.474+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037a90 con 0x7fcfe40412a0 2024-04-27T04:07:33.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:33.475+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8007220 con 0x7fcfe40412a0 2024-04-27T04:07:33.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:07:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:07:33.472+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 306 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:07:33.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:33.312+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:33.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:33.312+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:33.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:33.312+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:33.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:33.313+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:34.327 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:34.031+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:34.327 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:34.031+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:34.327 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:34.031+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:34.327 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:34.032+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:34.328 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:34.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:34.328 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:34.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:34.328 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:34.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:34.328 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:34.033+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:34.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:34.327+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:34.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:34.327+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:34.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:34.327+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:34.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:34.328+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:35.296 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:35.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:35.297 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:35.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:35.297 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:35.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:35.297 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:35.064+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:35.297 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:35.033+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:35.297 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:35.033+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:35.297 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:35.033+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:35.297 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:35.034+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:35.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:35.296+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:35.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:35.296+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:35.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:35.296+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:35.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:35.298+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:36.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:36.096+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:36.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:36.096+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:36.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:36.096+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:36.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:36.096+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:36.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:36.078+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:36.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:36.078+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:36.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:36.078+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:36.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:36.079+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:36.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:36.276+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:36.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:36.276+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:36.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:36.276+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:36.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:36.278+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:37.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:37.255+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:37.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:37.255+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:37.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:37.255+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:37.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:37.256+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:37.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:37.098+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:37.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:37.098+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:37.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:37.098+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:37.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:37.099+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:37.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:37.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:37.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:37.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:37.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:37.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:37.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:37.068+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:38.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:38.226+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:38.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:38.226+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:38.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:38.226+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:38.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:38.227+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:38.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:38.045+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:38.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:38.045+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:38.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:38.045+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:38.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:38.045+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:38.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:38.114+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:38.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:38.114+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:38.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:38.114+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:38.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:38.115+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.472+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:07:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.472+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:07:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.472+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8014940 tx=0x7fcfe8024000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:07:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.472+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8014940 tx=0x7fcfe8024000 comp rx=0 tx=0).stop 2024-04-27T04:07:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.472+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 msgr2=0x7fcfe404ef30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:07:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.472+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40412a0 0x7fcfe404ef30 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.472+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.472+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.472+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4050ee0 con 0x7fcfe40443d0 2024-04-27T04:07:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.472+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e6c0 con 0x7fcff8101eb0 2024-04-27T04:07:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.473+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:07:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.474+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:07:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.474+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.474+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044e90 con 0x7fcfe40443d0 2024-04-27T04:07:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.474+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fcff004df60 tx=0x7fcfe8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:07:38.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.475+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003a340 con 0x7fcfe40443d0 2024-04-27T04:07:38.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff00103a0 con 0x7fcfe40443d0 2024-04-27T04:07:38.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:38.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff002b220 con 0x7fcfe40443d0 2024-04-27T04:07:38.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:07:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:07:38.472+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 324 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:07:39.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:39.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:39.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:39.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:39.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:39.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:39.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:39.132+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:39.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:39.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:39.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:39.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:39.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:39.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:39.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:39.072+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:39.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:39.228+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:39.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:39.228+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:39.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:39.228+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:39.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:39.229+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:40.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:40.235+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:40.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:40.235+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:40.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:40.235+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:40.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:40.236+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:40.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:40.168+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:40.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:40.168+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:40.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:40.168+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:40.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:40.169+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:40.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:40.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:40.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:40.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:40.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:40.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:40.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:40.058+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:41.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:41.215+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:41.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:41.215+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:41.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:41.215+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:41.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:41.215+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:41.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:41.144+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:41.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:41.144+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:41.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:41.144+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:41.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:41.144+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:41.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:41.044+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:41.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:41.044+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:41.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:41.044+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:41.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:41.044+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:42.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:42.215+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:42.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:42.215+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:42.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:42.215+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:42.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:42.216+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:42.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:42.139+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:42.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:42.139+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:42.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:42.139+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:42.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:42.140+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:42.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:42.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:42.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:42.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:42.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:42.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:42.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:42.073+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:42.741+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:07:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:42.741+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:07:43.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:43.100+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:43.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:43.100+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:43.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:43.100+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:43.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:43.101+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:43.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:43.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:43.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:43.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:43.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:43.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:43.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:43.182+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:43.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:43.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:43.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:43.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:43.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:43.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:43.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:43.050+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:43.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.473+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:07:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.473+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:07:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.473+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fcff004df60 tx=0x7fcfe8009f90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:07:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.473+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fcff004df60 tx=0x7fcfe8009f90 comp rx=0 tx=0).stop 2024-04-27T04:07:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:07:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.473+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.473+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.473+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4043990 con 0x7fcfe404ef30 2024-04-27T04:07:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f310 con 0x7fcff8101eb0 2024-04-27T04:07:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.473+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:07:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.474+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:07:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.474+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:43.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.474+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe404e6c0 con 0x7fcfe404ef30 2024-04-27T04:07:43.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.474+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4014fa0 tx=0x7fcff001f120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:07:43.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.475+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe804f500 con 0x7fcfe404ef30 2024-04-27T04:07:43.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.475+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037a90 con 0x7fcfe404ef30 2024-04-27T04:07:43.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:43.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8007220 con 0x7fcfe404ef30 2024-04-27T04:07:43.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:07:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:07:43.473+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 340 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:07:44.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:44.143+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:44.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:44.143+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:44.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:44.143+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:44.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:44.143+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:44.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:44.150+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:44.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:44.150+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:44.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:44.150+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:44.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:44.152+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:44.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:44.043+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:44.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:44.043+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:44.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:44.043+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:44.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:44.044+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:45.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:45.163+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:45.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:45.163+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:45.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:45.163+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:45.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:45.164+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:45.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:45.134+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:45.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:45.134+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:45.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:45.134+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:45.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:45.134+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:45.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:45.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:45.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:45.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:45.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:45.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:45.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:45.034+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:46.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:46.177+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:46.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:46.177+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:46.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:46.177+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:46.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:46.178+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:46.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:46.190+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:46.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:46.190+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:46.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:46.190+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:46.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:46.192+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:46.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:46.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:46.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:46.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:46.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:46.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:46.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:46.066+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:47.148+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:47.148+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:47.148+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:47.149+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:47.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:47.104+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:47.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:47.104+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:47.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:47.104+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:47.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:47.105+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:47.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:47.171+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:47.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:47.171+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:47.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:47.171+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:47.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:47.171+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:48.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:48.173+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:48.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:48.173+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:48.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:48.173+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:48.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:48.174+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:48.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:48.174+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:48.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:48.174+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:48.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:48.174+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:48.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:48.175+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:48.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:48.083+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:48.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:48.083+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:48.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:48.083+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:48.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:48.083+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:48.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.473+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 msgr2=0x7fcfe40452a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:07:48.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.473+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 msgr2=0x7fcfe40452a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:07:48.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.473+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4014fa0 tx=0x7fcff001f120 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:07:48.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.473+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4014fa0 tx=0x7fcff001f120 comp rx=0 tx=0).stop 2024-04-27T04:07:48.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 msgr2=0x7fcfe40452a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:07:48.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.473+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:48.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.473+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:48.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.473+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:48.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcfe40443d0 2024-04-27T04:07:48.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e90 con 0x7fcff8101eb0 2024-04-27T04:07:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.474+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:07:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.475+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:07:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.475+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.475+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40457e0 con 0x7fcfe40443d0 2024-04-27T04:07:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.475+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800afe0 tx=0x7fcfe803a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:07:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8037a90 con 0x7fcfe40443d0 2024-04-27T04:07:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe80133a0 con 0x7fcfe40443d0 2024-04-27T04:07:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:48.477+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8007220 con 0x7fcfe40443d0 2024-04-27T04:07:48.885 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:07:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:07:48.473+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 357 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:07:49.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:49.167+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:49.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:49.167+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:49.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:49.167+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:49.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:49.168+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:49.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:49.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:49.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:49.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:49.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:49.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:49.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:49.165+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:49.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:49.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:49.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:49.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:49.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:49.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:49.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:49.064+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:50.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:50.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:50.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:50.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:50.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:50.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:50.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:50.205+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:50.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:50.136+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:50.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:50.136+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:50.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:50.136+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:50.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:50.138+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:50.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:50.014+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:50.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:50.014+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:50.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:50.014+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:50.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:50.015+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:51.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:51.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:51.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:51.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:51.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:51.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:51.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:51.064+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:51.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:51.166+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:51.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:51.166+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:51.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:51.166+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:51.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:51.168+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:51.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:51.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:51.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:51.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:51.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:51.205+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:51.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:51.205+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:52.126+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:52.126+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:52.126+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:52.127+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:52.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:52.213+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:52.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:52.213+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:52.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:52.213+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:52.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:52.213+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:52.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:52.085+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:52.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:52.085+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:52.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:52.085+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:52.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:52.086+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:53.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:53.091+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:53.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:53.091+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:53.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:53.091+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:53.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:53.092+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:53.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:53.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:53.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:53.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:53.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:53.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:53.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:53.143+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:53.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:53.167+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:53.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:53.167+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:53.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:53.167+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:53.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:53.167+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:53.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.474+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:07:53.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.474+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:07:53.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.474+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800afe0 tx=0x7fcfe803a040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:07:53.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.474+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800afe0 tx=0x7fcfe803a040 comp rx=0 tx=0).stop 2024-04-27T04:07:53.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.474+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:07:53.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.474+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:53.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.474+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:53.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.474+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:53.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.474+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcfe404ef30 2024-04-27T04:07:53.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.474+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044e70 con 0x7fcff8101eb0 2024-04-27T04:07:53.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.474+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:07:53.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.475+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:07:53.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.475+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:53.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.475+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40457e0 con 0x7fcfe404ef30 2024-04-27T04:07:53.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.475+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8007ed0 tx=0x7fcfe8008990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:07:53.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe80133a0 con 0x7fcfe404ef30 2024-04-27T04:07:53.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe404ef30 2024-04-27T04:07:53.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:53.477+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8007220 con 0x7fcfe404ef30 2024-04-27T04:07:53.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:07:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:07:53.474+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 374 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:07:54.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:54.067+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:54.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:54.067+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:54.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:54.067+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:54.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:54.068+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:54.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:54.143+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:54.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:54.143+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:54.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:54.143+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:54.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:54.144+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:54.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:54.182+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:54.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:54.182+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:54.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:54.182+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:54.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:54.183+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:55.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:55.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:55.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:55.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:55.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:55.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:55.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:55.082+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:55.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:55.135+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:55.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:55.135+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:55.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:55.135+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:55.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:55.135+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:55.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:55.172+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:55.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:55.172+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:55.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:55.172+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:55.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:55.173+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:56.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:56.108+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:56.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:56.108+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:56.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:56.108+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:56.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:56.109+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:56.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:56.161+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:56.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:56.161+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:56.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:56.161+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:56.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:56.162+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:56.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:56.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:56.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:56.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:56.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:56.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:56.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:56.051+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:57.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:57.198+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:57.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:57.198+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:57.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:57.198+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:57.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:57.199+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:57.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:57.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:57.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:57.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:57.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:57.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:57.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:57.078+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:57.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:57.127+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:57.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:57.127+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:57.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:57.127+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:57.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:57.127+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:58.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.100+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:07:58.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.100+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:07:58.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:58.196+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:58.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:58.196+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:58.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:58.196+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:58.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:58.197+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:07:58.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:58.107+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:58.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:58.107+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:58.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:58.107+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:58.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:58.108+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:58.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:58.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:58.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:58.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:58.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:58.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:58.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:58.069+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:58.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.473+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 msgr2=0x7fcfe40452a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:07:58.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.473+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 msgr2=0x7fcfe40452a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:07:58.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.473+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8007ed0 tx=0x7fcfe8008990 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:07:58.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.473+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8007ed0 tx=0x7fcfe8008990 comp rx=0 tx=0).stop 2024-04-27T04:07:58.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 msgr2=0x7fcfe40452a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:07:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.473+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.473+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.473+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:07:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051c40 con 0x7fcfe40443d0 2024-04-27T04:07:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcff8101eb0 2024-04-27T04:07:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.474+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:07:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.474+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:07:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.474+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:07:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.474+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044e70 con 0x7fcfe40443d0 2024-04-27T04:07:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.475+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fcff0024fe0 tx=0x7fcfe80547c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:07:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.475+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003a340 con 0x7fcfe40443d0 2024-04-27T04:07:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.475+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff00103a0 con 0x7fcfe40443d0 2024-04-27T04:07:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:07:58.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff002b220 con 0x7fcfe40443d0 2024-04-27T04:07:58.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:07:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:07:58.473+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 392 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:07:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:59.123+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:59.123+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:59.123+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:07:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:07:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:07:59.123+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:59.111+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:59.111+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:59.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:59.111+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:07:59.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:07:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:07:59.111+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:07:59.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:59.163+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:59.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:59.163+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:59.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:59.163+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:07:59.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:07:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:07:59.163+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:00.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:00.177+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:00.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:00.177+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:00.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:00.177+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:00.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:00.177+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:00.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:00.119+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:00.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:00.119+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:00.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:00.119+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:00.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:00.120+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:00.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:00.105+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:00.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:00.105+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:00.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:00.105+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:00.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:00.106+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:01.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:01.138+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:01.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:01.139+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:01.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:01.139+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:01.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:01.140+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:01.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:01.208+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:01.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:01.208+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:01.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:01.208+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:01.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:01.209+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:01.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:01.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:01.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:01.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:01.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:01.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:01.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:01.111+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:02.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:02.138+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:02.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:02.138+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:02.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:02.138+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:02.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:02.138+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:02.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:02.144+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:02.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:02.144+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:02.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:02.144+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:02.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:02.144+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:02.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:02.240+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:02.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:02.240+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:02.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:02.240+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:02.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:02.240+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:03.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:03.264+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:03.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:03.264+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:03.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:03.264+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:03.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:03.265+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:03.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:03.099+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:03.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:03.099+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:03.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:03.099+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:03.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:03.100+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:03.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:03.182+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:03.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:03.182+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:03.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:03.182+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:03.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:03.182+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:03.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.473+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:08:03.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.473+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:08:03.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.473+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fcff0024fe0 tx=0x7fcfe80547c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:08:03.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.473+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fcff0024fe0 tx=0x7fcfe80547c0 comp rx=0 tx=0).stop 2024-04-27T04:08:03.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.473+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 msgr2=0x7fcfe40504c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:08:03.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.473+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40443d0 0x7fcfe40504c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:03.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.474+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:03.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.474+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:03.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.474+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcfe404ef30 2024-04-27T04:08:03.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.474+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051c40 con 0x7fcff8101eb0 2024-04-27T04:08:03.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.474+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:08:03.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.474+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:08:03.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.474+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:03.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.474+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044dc0 con 0x7fcfe404ef30 2024-04-27T04:08:03.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.475+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8021ff0 tx=0x7fcff0036b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:08:03.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.475+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe404ef30 2024-04-27T04:08:03.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.475+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe404ef30 2024-04-27T04:08:03.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:03.477+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8054450 con 0x7fcfe404ef30 2024-04-27T04:08:03.753 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:08:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:08:03.474+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 408 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:08:04.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:04.180+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:04.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:04.180+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:04.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:04.180+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:04.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:04.181+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:04.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:04.296+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:04.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:04.296+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:04.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:04.296+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:04.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:04.296+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:04.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:04.138+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:04.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:04.138+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:04.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:04.138+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:04.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:04.140+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:05.161+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:05.161+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:05.161+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:05.162+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:05.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:05.284+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:05.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:05.284+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:05.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:05.284+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:05.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:05.285+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:05.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:05.115+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:05.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:05.115+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:05.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:05.115+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:05.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:05.116+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:06.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:06.330+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:06.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:06.330+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:06.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:06.330+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:06.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:06.331+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:06.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:06.134+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:06.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:06.134+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:06.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:06.134+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:06.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:06.134+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:06.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:06.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:06.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:06.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:06.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:06.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:06.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:06.189+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:07.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:07.356+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:07.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:07.356+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:07.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:07.356+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:07.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:07.356+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:07.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:07.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:07.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:07.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:07.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:07.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:07.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:07.214+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:07.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:07.150+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:07.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:07.150+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:07.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:07.150+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:07.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:07.150+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:08.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:08.166+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:08.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:08.166+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:08.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:08.166+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:08.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:08.167+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:08.358+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:08.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:08.358+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:08.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:08.358+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:08.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:08.359+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:08.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:08.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:08.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:08.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:08.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:08.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:08.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:08.176+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:08.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.475+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 msgr2=0x7fcfe40452a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:08:08.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.475+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 msgr2=0x7fcfe40452a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:08:08.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.475+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8021ff0 tx=0x7fcff0036b10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:08:08.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.475+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8021ff0 tx=0x7fcff0036b10 comp rx=0 tx=0).stop 2024-04-27T04:08:08.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.475+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 msgr2=0x7fcfe40452a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:08:08.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.475+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:08.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.475+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40504c0 0x7fcfe40457e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:08.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.475+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:08.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.475+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcfe40504c0 2024-04-27T04:08:08.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.475+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044dc0 con 0x7fcff8101eb0 2024-04-27T04:08:08.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.475+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40504c0 0x7fcfe40457e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:08:08.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.476+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:08:08.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.476+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:08.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.476+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044f00 con 0x7fcfe40504c0 2024-04-27T04:08:08.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.477+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40504c0 0x7fcfe40457e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8017dd0 tx=0x7fcfe8037c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:08:08.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.477+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe40504c0 2024-04-27T04:08:08.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.478+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804bde0 con 0x7fcfe40504c0 2024-04-27T04:08:08.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:08.479+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8054450 con 0x7fcfe40504c0 2024-04-27T04:08:08.813 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:08:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:08:08.475+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 425 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:08:09.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:09.128+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:09.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:09.128+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:09.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:09.128+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:09.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:09.129+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:09.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:09.153+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:09.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:09.153+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:09.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:09.153+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:09.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:09.154+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:09.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:09.380+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:09.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:09.380+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:09.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:09.380+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:09.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:09.381+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:10.397 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:10.099+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:10.397 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:10.099+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:10.397 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:10.099+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:10.397 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:10.100+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:10.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:10.132+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:10.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:10.132+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:10.398 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:10.132+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:10.398 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:10.133+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:10.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:10.397+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:10.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:10.397+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:10.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:10.397+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:10.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:10.398+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:11.398 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:11.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:11.399 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:11.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:11.399 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:11.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:11.399 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:11.105+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:11.399 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:11.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:11.399 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:11.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:11.399 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:11.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:11.399 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:11.068+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:11.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:11.399+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:11.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:11.399+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:11.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:11.399+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:11.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:11.400+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:12.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:12.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:12.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:12.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:12.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:12.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:12.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:12.059+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:12.377 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:12.079+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:12.377 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:12.079+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:12.378 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:12.079+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:12.378 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:12.080+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:12.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:12.377+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:12.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:12.377+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:12.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:12.377+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:12.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:12.378+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:13.418 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:13.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:13.418 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:13.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:13.418 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:13.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:13.418 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:13.051+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:13.419 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:13.080+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:13.419 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:13.080+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:13.419 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:13.080+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:13.419 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:13.082+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:13.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.461+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:08:13.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.461+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:08:13.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.475+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40504c0 msgr2=0x7fcfe40457e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:08:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.475+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40504c0 msgr2=0x7fcfe40457e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:08:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.475+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40504c0 0x7fcfe40457e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8017dd0 tx=0x7fcfe8037c30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:08:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.475+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40504c0 0x7fcfe40457e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8017dd0 tx=0x7fcfe8037c30 comp rx=0 tx=0).stop 2024-04-27T04:08:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40504c0 msgr2=0x7fcfe40457e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:08:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.476+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40504c0 0x7fcfe40457e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.476+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.476+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcfe404ef30 2024-04-27T04:08:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044dc0 con 0x7fcff8101eb0 2024-04-27T04:08:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.476+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:08:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.477+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:08:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.477+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.477+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044f00 con 0x7fcfe404ef30 2024-04-27T04:08:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.477+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80012f0 tx=0x7fcfe8007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:08:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.478+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe404ef30 2024-04-27T04:08:13.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.478+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe404ef30 2024-04-27T04:08:13.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:13.479+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8054450 con 0x7fcfe404ef30 2024-04-27T04:08:13.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:08:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:08:13.476+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 442 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:08:13.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:13.418+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:13.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:13.419+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:13.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:13.419+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:13.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:13.420+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:14.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:14.052+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:14.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:14.052+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:14.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:14.052+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:14.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:14.053+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:14.018+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:14.018+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:14.018+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:14.018+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:14.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:14.434+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:14.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:14.434+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:14.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:14.434+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:14.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:14.435+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:15.393 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:15.102+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:15.393 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:15.102+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:15.393 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:15.102+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:15.393 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:15.103+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:15.394 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:14.978+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:15.394 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:14.978+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:15.394 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:14.978+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:15.394 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:14.979+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:15.395 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:15.393+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:15.395 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:15.393+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:15.395 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:15.393+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:15.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:15.394+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:16.416 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:16.098+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:16.416 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:16.098+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:16.416 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:16.098+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:16.416 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:16.098+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:16.417 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:15.946+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:16.417 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:15.946+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:16.417 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:15.946+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:16.417 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:15.947+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:16.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:16.416+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:16.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:16.416+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:16.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:16.416+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:16.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:16.417+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:17.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:17.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:17.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:17.082+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:17.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:16.941+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:17.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:16.941+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:17.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:16.941+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:17.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:16.942+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:17.893 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:17.439+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:17.893 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:17.439+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:17.893 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:17.439+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:17.893 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:17.439+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:18.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:18.125+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:18.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:18.125+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:18.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:18.125+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:18.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:18.126+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:18.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:17.893+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:18.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:17.893+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:18.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:17.893+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:18.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:17.895+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:18.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.476+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 msgr2=0x7fcfe40452a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:08:18.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.476+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 msgr2=0x7fcfe40452a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:08:18.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.476+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80012f0 tx=0x7fcfe8007f00 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:08:18.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.476+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80012f0 tx=0x7fcfe8007f00 comp rx=0 tx=0).stop 2024-04-27T04:08:18.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 msgr2=0x7fcfe40452a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:08:18.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.476+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:18.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.476+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:18.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.476+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40457e0 0x7fcfe4045bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:18.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051c40 con 0x7fcfe40457e0 2024-04-27T04:08:18.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcff8101eb0 2024-04-27T04:08:18.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.476+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40457e0 0x7fcfe4045bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:08:18.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.477+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:08:18.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.477+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:18.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.478+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044dc0 con 0x7fcfe40457e0 2024-04-27T04:08:18.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.478+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40457e0 0x7fcfe4045bc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fcff004ef10 tx=0x7fcfe800b300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:08:18.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.479+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0011fc0 con 0x7fcfe40457e0 2024-04-27T04:08:18.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.479+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe40457e0 2024-04-27T04:08:18.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:18.481+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff002b220 con 0x7fcfe40457e0 2024-04-27T04:08:18.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:08:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:08:18.476+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 460 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:08:18.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:18.426+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:18.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:18.426+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:18.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:18.426+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:18.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:18.427+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:19.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:18.937+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:19.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:18.937+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:19.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:18.937+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:19.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:18.938+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:19.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:19.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:19.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:19.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:19.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:19.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:19.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:19.093+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:19.895 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:19.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:19.895 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:19.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:19.895 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:19.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:19.895 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:19.465+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:20.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:20.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:20.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:20.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:20.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:20.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:20.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:20.130+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:20.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:19.895+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:20.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:19.895+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:20.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:19.895+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:20.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:19.897+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:20.885 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:20.506+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:20.886 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:20.506+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:20.886 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:20.506+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:20.886 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:20.507+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:21.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:20.886+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:21.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:20.886+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:21.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:20.886+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:21.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:20.887+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:21.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:21.121+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:21.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:21.121+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:21.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:21.121+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:21.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:21.122+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:21.894 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:21.461+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:21.894 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:21.461+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:21.894 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:21.461+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:21.894 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:21.462+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:22.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:22.125+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:22.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:22.125+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:22.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:22.125+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:22.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:22.126+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:22.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:21.894+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:22.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:21.894+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:22.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:21.894+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:22.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:21.895+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:22.879 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:22.488+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:22.879 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:22.488+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:22.879 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:22.488+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:22.879 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:22.489+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:23.165 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:22.880+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:23.166 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:22.880+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:23.166 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:22.880+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:23.166 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:22.881+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:23.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:23.166+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:23.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:23.166+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:23.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:23.166+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:23.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:23.167+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:23.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.476+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40457e0 msgr2=0x7fcfe4045bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:08:23.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.476+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40457e0 msgr2=0x7fcfe4045bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:08:23.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.476+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40457e0 0x7fcfe4045bc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fcff004ef10 tx=0x7fcfe800b300 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:08:23.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.476+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40457e0 0x7fcfe4045bc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fcff004ef10 tx=0x7fcfe800b300 comp rx=0 tx=0).stop 2024-04-27T04:08:23.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40457e0 msgr2=0x7fcfe4045bc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:08:23.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.476+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40457e0 0x7fcfe4045bc0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:23.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.476+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:23.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.476+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:23.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40519a0 con 0x7fcfe404ef30 2024-04-27T04:08:23.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.476+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcff8101eb0 2024-04-27T04:08:23.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.477+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:08:23.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.477+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:08:23.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.477+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:23.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.477+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044dc0 con 0x7fcfe404ef30 2024-04-27T04:08:23.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.478+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40112c0 tx=0x7fcff0012b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:08:23.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.479+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0011fc0 con 0x7fcfe404ef30 2024-04-27T04:08:23.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.479+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0025cc0 con 0x7fcfe404ef30 2024-04-27T04:08:23.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:23.480+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff002b220 con 0x7fcfe404ef30 2024-04-27T04:08:23.844 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:08:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:08:23.476+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 476 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:08:23.845 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:23.529+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:23.845 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:23.529+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:23.845 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:23.529+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:23.845 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:23.530+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:24.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:23.845+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:24.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:23.845+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:24.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:23.845+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:24.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:23.846+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:24.561 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:24.209+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:24.562 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:24.209+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:24.562 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:24.209+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:24.562 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:24.209+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:24.863 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:24.562+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:24.863 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:24.562+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:24.863 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:24.562+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:24.863 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:24.563+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:25.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:24.864+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:25.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:24.864+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:25.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:24.864+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:25.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:24.864+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:25.525 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:25.216+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:25.526 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:25.216+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:25.526 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:25.216+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:25.526 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:25.217+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:25.526 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:25.526+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:25.526 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:25.526+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:25.526 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:25.526+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:25.863 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:25.527+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:26.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:25.864+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:26.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:25.864+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:26.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:25.864+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:26.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:25.865+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:26.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:26.257+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:26.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:26.257+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:26.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:26.257+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:26.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:26.258+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:26.482+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:26.482+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:26.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:26.482+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:26.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:26.483+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:27.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:26.866+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:27.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:26.866+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:27.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:26.866+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:27.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:26.867+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:27.523 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:27.220+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:27.524 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:27.220+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:27.524 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:27.220+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:27.524 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:27.221+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:27.908 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:27.524+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:27.908 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:27.524+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:27.908 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:27.524+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:27.908 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:27.525+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:28.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:27.909+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:28.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:27.909+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:28.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:27.909+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:28.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:27.909+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:28.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.477+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 msgr2=0x7fcfe40452a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:08:28.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.477+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 msgr2=0x7fcfe40452a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:08:28.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.477+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40112c0 tx=0x7fcff0012b20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:08:28.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.477+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40112c0 tx=0x7fcff0012b20 comp rx=0 tx=0).stop 2024-04-27T04:08:28.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.477+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 msgr2=0x7fcfe40452a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:08:28.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.477+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe404ef30 0x7fcfe40452a0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:28.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.477+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 0x7fcfe4045fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:28.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.477+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:28.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.477+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcfe4045bc0 2024-04-27T04:08:28.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.477+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044dc0 con 0x7fcff8101eb0 2024-04-27T04:08:28.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.477+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 0x7fcfe4045fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:08:28.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.477+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:08:28.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.477+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:28.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.478+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044f00 con 0x7fcfe4045bc0 2024-04-27T04:08:28.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.478+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 0x7fcfe4045fa0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8014940 tx=0x7fcff002d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:08:28.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.478+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800cea0 con 0x7fcfe4045bc0 2024-04-27T04:08:28.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.478+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8036ac0 con 0x7fcfe4045bc0 2024-04-27T04:08:28.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.480+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe80083a0 con 0x7fcfe4045bc0 2024-04-27T04:08:28.547 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:08:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:08:28.477+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 493 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:08:28.547 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:28.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:28.547 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:28.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:28.548 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:28.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:28.548 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:28.259+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:28.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.821+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:08:28.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:28.821+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:08:28.820 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:28.548+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:28.821 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:28.548+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:28.821 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:28.548+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:28.821 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:28.549+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:29.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:28.928+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:29.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:28.928+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:29.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:28.928+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:29.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:28.929+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:29.519+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:29.519+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:29.519+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:29.520+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:29.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:29.275+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:29.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:29.275+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:29.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:29.275+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:29.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:29.276+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:30.261 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:29.924+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:30.261 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:29.924+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:30.261 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:29.924+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:30.261 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:29.925+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:30.515 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:30.261+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:30.515 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:30.261+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:30.515 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:30.261+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:30.515 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:30.262+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:30.920 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:30.515+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:30.920 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:30.515+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:30.920 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:30.515+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:30.920 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:30.515+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:31.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:30.920+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:31.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:30.920+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:31.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:30.920+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:31.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:30.921+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:31.469 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:31.231+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:31.469 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:31.231+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:31.469 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:31.231+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:31.469 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:31.231+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:31.871 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:31.469+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:31.872 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:31.469+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:31.872 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:31.469+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:31.872 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:31.470+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:32.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:31.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:32.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:31.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:32.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:31.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:32.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:31.873+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:32.490 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:32.211+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:32.491 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:32.211+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:32.491 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:32.211+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:32.491 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:32.211+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:32.833 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:32.491+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:32.833 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:32.491+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:32.833 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:32.491+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:32.833 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:32.492+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:33.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:32.833+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:33.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:32.833+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:33.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:32.833+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:33.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:32.834+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:33.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.477+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 msgr2=0x7fcfe4045fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-27T04:08:33.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.477+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 msgr2=0x7fcfe4045fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:08:33.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.477+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 0x7fcfe4045fa0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8014940 tx=0x7fcff002d040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:08:33.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.477+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 0x7fcfe4045fa0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8014940 tx=0x7fcff002d040 comp rx=0 tx=0).stop 2024-04-27T04:08:33.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.478+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 msgr2=0x7fcfe4045fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:08:33.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.478+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 0x7fcfe4045fa0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:33.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.478+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 0x7fcfe4053f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:33.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.478+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:33.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.478+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4045680 con 0x7fcfe40452a0 2024-04-27T04:08:33.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.478+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcff8101eb0 2024-04-27T04:08:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.479+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 0x7fcfe4053f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:08:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.479+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:08:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.479+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.479+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044dc0 con 0x7fcfe40452a0 2024-04-27T04:08:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.480+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 0x7fcfe4053f20 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fcff0012120 tx=0x7fcfe804f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:08:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.481+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800cea0 con 0x7fcfe40452a0 2024-04-27T04:08:33.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.481+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe40452a0 2024-04-27T04:08:33.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:33.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe80083a0 con 0x7fcfe40452a0 2024-04-27T04:08:33.524 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:08:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:08:33.478+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 510 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:08:33.524 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:33.245+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:33.525 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:33.245+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:33.525 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:33.245+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:33.525 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:33.247+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:33.525 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:33.447+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:33.525 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:33.447+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:33.525 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:33.447+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:33.525 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:33.448+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:34.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:33.860+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:34.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:33.860+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:34.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:33.860+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:34.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:33.860+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:34.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:34.453+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:34.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:34.453+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:34.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:34.453+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:34.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:34.454+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:34.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:34.294+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:34.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:34.294+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:34.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:34.294+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:34.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:34.295+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:34.899+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:34.899+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:34.899+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:34.900+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:35.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:35.416+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:35.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:35.416+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:35.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:35.416+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:35.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:35.417+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:35.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:35.281+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:35.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:35.281+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:35.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:35.281+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:35.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:35.281+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:36.266 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:35.935+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:36.266 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:35.935+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:36.266 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:35.935+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:36.266 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:35.935+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:36.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:36.267+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:36.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:36.267+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:36.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:36.267+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:36.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:36.268+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:36.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:36.373+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:36.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:36.373+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:36.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:36.373+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:36.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:36.374+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:37.305 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:36.965+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:37.305 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:36.965+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:37.305 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:36.965+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:37.305 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:36.966+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:37.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:37.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:37.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:37.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:37.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:37.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:37.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:37.306+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:37.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:37.340+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:37.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:37.340+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:37.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:37.340+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:37.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:37.341+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:38.326 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:37.971+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:38.326 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:37.971+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:38.326 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:37.971+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:38.326 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:37.972+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.478+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 msgr2=0x7fcfe4053f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:08:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.478+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 msgr2=0x7fcfe4053f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:08:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.478+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 0x7fcfe4053f20 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fcff0012120 tx=0x7fcfe804f6f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:08:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.478+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 0x7fcfe4053f20 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fcff0012120 tx=0x7fcfe804f6f0 comp rx=0 tx=0).stop 2024-04-27T04:08:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.478+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 msgr2=0x7fcfe4053f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:08:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.478+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 0x7fcfe4053f20 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.478+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.478+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 0x7fcfe4045fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.478+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcfe4045bc0 2024-04-27T04:08:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.478+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044dc0 con 0x7fcff8101eb0 2024-04-27T04:08:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.479+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 0x7fcfe4045fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:08:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.479+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:08:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.479+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.479+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044f00 con 0x7fcfe4045bc0 2024-04-27T04:08:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.480+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 0x7fcfe4045fa0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fcff0010e70 tx=0x7fcfe8049040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:08:38.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.481+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0011fc0 con 0x7fcfe4045bc0 2024-04-27T04:08:38.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.481+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe4045bc0 2024-04-27T04:08:38.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:38.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff002b220 con 0x7fcfe4045bc0 2024-04-27T04:08:38.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:08:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:08:38.478+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 528 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:08:38.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:38.326+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:38.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:38.326+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:38.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:38.326+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:38.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:38.327+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:38.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:38.376+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:38.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:38.376+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:38.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:38.376+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:38.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:38.377+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:39.345 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:39.001+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:39.345 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:39.001+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:39.345 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:39.001+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:39.345 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:39.002+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:39.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:39.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:39.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:39.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:39.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:39.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:39.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:39.347+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:39.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:39.382+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:39.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:39.382+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:39.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:39.382+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:39.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:39.383+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:40.319 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:40.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:40.319 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:40.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:40.319 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:40.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:40.319 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:40.034+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:40.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:40.412+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:40.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:40.412+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:40.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:40.412+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:40.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:40.413+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:40.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:40.319+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:40.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:40.319+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:40.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:40.319+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:40.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:40.320+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:41.311 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:41.008+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:41.311 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:41.008+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:41.311 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:41.008+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:41.311 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:41.009+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:41.569 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:41.311+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:41.569 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:41.311+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:41.569 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:41.311+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:41.569 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:41.312+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:41.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:41.409+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:41.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:41.409+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:41.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:41.409+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:41.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:41.410+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:42.279 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:41.975+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:42.279 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:41.975+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:42.279 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:41.975+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:42.279 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:41.976+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:42.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:42.279+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:42.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:42.279+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:42.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:42.279+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:42.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:42.281+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:42.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:42.437+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:42.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:42.437+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:42.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:42.437+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:42.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:42.438+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:43.319 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:43.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:43.319 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:43.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:43.319 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:43.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:43.319 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:43.008+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:43.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.478+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 msgr2=0x7fcfe4045fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:08:43.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.478+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 msgr2=0x7fcfe4045fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:08:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.478+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 0x7fcfe4045fa0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fcff0010e70 tx=0x7fcfe8049040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:08:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.478+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 0x7fcfe4045fa0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fcff0010e70 tx=0x7fcfe8049040 comp rx=0 tx=0).stop 2024-04-27T04:08:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.478+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 msgr2=0x7fcfe4045fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:08:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.478+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045bc0 0x7fcfe4045fa0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.478+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.478+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 0x7fcfe4053f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.478+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f470 con 0x7fcfe40452a0 2024-04-27T04:08:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.478+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044dc0 con 0x7fcff8101eb0 2024-04-27T04:08:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.479+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 0x7fcfe4053f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:08:43.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.479+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:08:43.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.479+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:43.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.479+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044f00 con 0x7fcfe40452a0 2024-04-27T04:08:43.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.479+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 0x7fcfe4053f20 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40471e0 tx=0x7fcff0006210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:08:43.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.480+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe40452a0 2024-04-27T04:08:43.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.480+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe40452a0 2024-04-27T04:08:43.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:43.481+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff002b220 con 0x7fcfe40452a0 2024-04-27T04:08:43.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:08:43.478+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 579 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:08:43.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:43.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:43.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:43.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:43.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:43.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:43.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:43.466+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:43.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:43.319+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:43.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:43.319+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:43.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:43.319+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:43.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:43.320+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:44.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:44.181+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:08:44.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:44.181+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:08:44.272 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:43.985+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:44.273 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:43.985+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:44.273 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:43.985+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:44.273 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:43.985+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:44.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:44.273+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:44.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:44.273+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:44.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:44.273+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:44.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:44.274+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:44.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:44.425+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:44.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:44.425+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:44.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:44.425+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:44.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:44.426+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:45.303 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:45.034+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:45.303 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:45.034+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:45.303 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:45.034+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:45.303 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:45.035+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:45.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:45.449+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:45.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:45.449+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:45.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:45.449+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:45.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:45.450+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:45.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:45.303+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:45.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:45.303+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:45.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:45.303+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:45.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:45.304+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:46.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:46.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:46.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:46.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:46.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:46.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:46.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:46.050+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:46.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:46.285+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:46.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:46.285+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:46.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:46.285+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:46.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:46.286+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:46.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:46.469+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:46.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:46.469+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:46.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:46.469+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:46.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:46.470+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:47.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:47.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:47.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:47.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:47.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:47.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:47.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:47.060+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:47.269+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:47.269+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:47.269+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:47.270+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:47.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:47.444+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:47.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:47.444+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:47.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:47.444+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:47.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:47.445+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:48.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:48.270+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:48.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:48.271+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:48.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:48.271+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:48.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:48.271+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:48.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:48.099+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:48.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:48.099+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:48.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:48.099+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:48.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:48.101+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:48.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.479+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 msgr2=0x7fcfe4053f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:08:48.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.479+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 msgr2=0x7fcfe4053f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:08:48.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.479+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 0x7fcfe4053f20 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40471e0 tx=0x7fcff0006210 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:08:48.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.479+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 0x7fcfe4053f20 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40471e0 tx=0x7fcff0006210 comp rx=0 tx=0).stop 2024-04-27T04:08:48.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.479+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 msgr2=0x7fcfe4053f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:08:48.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.479+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40452a0 0x7fcfe4053f20 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:48.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.479+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:48.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.479+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045fa0 0x7fcfe4054460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:48.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.479+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4046380 con 0x7fcfe4045fa0 2024-04-27T04:08:48.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.479+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcff8101eb0 2024-04-27T04:08:48.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.480+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045fa0 0x7fcfe4054460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:08:48.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.480+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:08:48.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.480+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:48.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.480+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044dc0 con 0x7fcfe4045fa0 2024-04-27T04:08:48.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.481+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045fa0 0x7fcfe4054460 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40471b0 tx=0x7fcff004dc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:08:48.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.481+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe4045fa0 2024-04-27T04:08:48.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe4045fa0 2024-04-27T04:08:48.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:48.483+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4045fa0 2024-04-27T04:08:48.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:08:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:08:48.479+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 633 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:08:48.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:48.425+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:48.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:48.425+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:48.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:48.425+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:48.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:48.426+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:49.406 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:49.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:49.406 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:49.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:49.406 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:49.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:49.407 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:49.111+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:49.407 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:49.306+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:49.407 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:49.306+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:49.407 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:49.306+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:49.407 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:49.307+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:49.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:49.407+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:49.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:49.407+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:49.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:49.407+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:49.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:49.408+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:50.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:50.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:50.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:50.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:50.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:50.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:50.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:50.079+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:50.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:50.313+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:50.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:50.313+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:50.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:50.313+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:50.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:50.315+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:50.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:50.421+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:50.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:50.421+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:50.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:50.421+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:50.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:50.422+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:51.346 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:51.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:51.346 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:51.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:51.346 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:51.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:51.346 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:51.074+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:51.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:51.402+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:51.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:51.402+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:51.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:51.402+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:51.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:51.403+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:51.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:51.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:51.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:51.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:51.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:51.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:51.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:51.347+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:52.365 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:52.055+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:52.365 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:52.055+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:52.365 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:52.055+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:52.366 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:52.056+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:52.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:52.366+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:52.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:52.366+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:52.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:52.366+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:52.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:52.366+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:52.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:52.399+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:52.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:52.399+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:52.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:52.399+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:52.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:52.400+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:53.369 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:53.084+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:53.369 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:53.084+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:53.369 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:53.084+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:53.369 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:53.086+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.479+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045fa0 msgr2=0x7fcfe4054460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:08:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.479+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045fa0 msgr2=0x7fcfe4054460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:08:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.479+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045fa0 0x7fcfe4054460 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40471b0 tx=0x7fcff004dc50 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:08:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.479+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045fa0 0x7fcfe4054460 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40471b0 tx=0x7fcff004dc50 comp rx=0 tx=0).stop 2024-04-27T04:08:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.479+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045fa0 msgr2=0x7fcfe4054460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:08:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.479+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4045fa0 0x7fcfe4054460 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.479+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.479+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4053f20 0x7fcfe40549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.479+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4046380 con 0x7fcfe4053f20 2024-04-27T04:08:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.479+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcff8101eb0 2024-04-27T04:08:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.480+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4053f20 0x7fcfe40549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:08:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.480+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:08:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.480+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.480+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044dc0 con 0x7fcfe4053f20 2024-04-27T04:08:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.481+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4053f20 0x7fcfe40549a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400dcc0 tx=0x7fcff0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:08:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe4053f20 2024-04-27T04:08:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff002aca0 con 0x7fcfe4053f20 2024-04-27T04:08:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:53.483+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4053f20 2024-04-27T04:08:53.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:08:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:08:53.479+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 680 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:08:53.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:53.392+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:53.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:53.392+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:53.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:53.392+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:53.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:53.393+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:53.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:53.369+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:53.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:53.369+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:53.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:53.369+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:53.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:53.370+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:54.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:54.124+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:54.393 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:54.124+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:54.393 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:54.124+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:54.393 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:54.125+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:54.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:54.419+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:54.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:54.419+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:54.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:54.419+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:54.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:54.420+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:54.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:54.393+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:54.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:54.393+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:54.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:54.393+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:54.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:54.396+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:55.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:55.391+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:55.391+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:55.391+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:55.392+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:55.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:55.164+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:55.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:55.164+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:55.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:55.164+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:55.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:55.165+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:55.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:55.388+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:55.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:55.388+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:55.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:55.388+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:55.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:55.388+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:56.468 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:56.187+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:56.468 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:56.187+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:56.469 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:56.187+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:56.469 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:56.188+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:56.469 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:56.404+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:56.469 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:56.404+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:56.469 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:56.404+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:56.470 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:56.405+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:56.470 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:56.379+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:56.470 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:56.379+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:56.470 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:56.379+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:56.470 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:56.380+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:57.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:57.435+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:57.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:57.435+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:57.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:57.435+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:57.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:57.436+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:57.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:57.336+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:57.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:57.336+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:57.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:57.336+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:57.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:57.337+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:57.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:57.212+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:57.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:57.212+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:57.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:57.212+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:57.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:57.212+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:58.445 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:58.340+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:58.445 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:58.340+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:58.445 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:58.340+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:58.445 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:58.341+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:58.445 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:58.189+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:58.445 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:58.189+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:58.445 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:58.189+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:58.446 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:58.189+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:58.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.479+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4053f20 msgr2=0x7fcfe40549a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:08:58.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.480+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4053f20 msgr2=0x7fcfe40549a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:08:58.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.480+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4053f20 0x7fcfe40549a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400dcc0 tx=0x7fcff0000bc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:08:58.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.480+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4053f20 0x7fcfe40549a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400dcc0 tx=0x7fcff0000bc0 comp rx=0 tx=0).stop 2024-04-27T04:08:58.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.480+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4053f20 msgr2=0x7fcfe40549a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:08:58.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.480+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4053f20 0x7fcfe40549a0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:58.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.480+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054340 0x7fcfe4054ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:58.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.480+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:08:58.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.480+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4046380 con 0x7fcfe4054340 2024-04-27T04:08:58.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.480+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcff8101eb0 2024-04-27T04:08:58.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.480+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054340 0x7fcfe4054ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:08:58.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.481+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:08:58.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.481+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:08:58.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.481+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4054720 con 0x7fcfe4054340 2024-04-27T04:08:58.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.482+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054340 0x7fcfe4054ee0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40176c0 tx=0x7fcff00339d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:08:58.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4054340 2024-04-27T04:08:58.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8036300 con 0x7fcfe4054340 2024-04-27T04:08:58.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:58.483+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe80083a0 con 0x7fcfe4054340 2024-04-27T04:08:58.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:08:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:08:58.480+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 741 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:08:58.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:58.445+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:58.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:58.445+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:58.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:58.445+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:58.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:58.446+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:08:59.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:59.541+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:08:59.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:08:59.541+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:08:59.540 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:59.354+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:59.541 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:59.354+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:59.541 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:59.354+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:08:59.541 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:08:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:08:59.355+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:59.541 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:59.184+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:59.541 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:59.184+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:59.541 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:59.184+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:08:59.541 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:08:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:08:59.184+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:08:59.542 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:59.433+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:59.542 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:59.433+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:59.542 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:59.433+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:08:59.542 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:08:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:08:59.434+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:00.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:00.335+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:00.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:00.335+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:00.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:00.335+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:00.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:00.336+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:00.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:00.441+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:00.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:00.441+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:00.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:00.441+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:00.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:00.442+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:00.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:00.204+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:00.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:00.204+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:00.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:00.204+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:00.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:00.206+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:01.464 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:01.378+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:01.465 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:01.378+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:01.465 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:01.378+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:01.465 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:01.379+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:01.465 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:01.186+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:01.465 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:01.186+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:01.465 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:01.186+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:01.465 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:01.187+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:01.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:01.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:01.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:01.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:01.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:01.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:01.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:01.466+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:02.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:02.229+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:02.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:02.229+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:02.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:02.229+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:02.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:02.230+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:02.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:02.340+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:02.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:02.340+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:02.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:02.340+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:02.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:02.341+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:02.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:02.442+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:02.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:02.442+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:02.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:02.442+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:02.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:02.443+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:03.468 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:03.348+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:03.468 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:03.348+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:03.468 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:03.348+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:03.468 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:03.349+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:03.469 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:03.201+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:03.469 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:03.201+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:03.469 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:03.201+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:03.469 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:03.203+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.480+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054340 msgr2=0x7fcfe4054ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:09:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.480+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054340 msgr2=0x7fcfe4054ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:09:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.480+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054340 0x7fcfe4054ee0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40176c0 tx=0x7fcff00339d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:09:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.480+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054340 0x7fcfe4054ee0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40176c0 tx=0x7fcff00339d0 comp rx=0 tx=0).stop 2024-04-27T04:09:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.480+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054340 msgr2=0x7fcfe4054ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:09:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.480+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054340 0x7fcfe4054ee0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.480+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.480+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.480+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcfe4054720 2024-04-27T04:09:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.480+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044da0 con 0x7fcff8101eb0 2024-04-27T04:09:03.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.481+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:09:03.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.481+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:09:03.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.481+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:03.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.481+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044ee0 con 0x7fcfe4054720 2024-04-27T04:09:03.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.482+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055420 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fcff0012ef0 tx=0x7fcfe80465e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:09:03.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003a340 con 0x7fcfe4054720 2024-04-27T04:09:03.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.483+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff000e400 con 0x7fcfe4054720 2024-04-27T04:09:03.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:03.483+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4054720 2024-04-27T04:09:03.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:09:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:09:03.480+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 802 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:09:03.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:03.469+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:03.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:03.469+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:03.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:03.469+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:03.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:03.469+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:04.492 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:04.234+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:04.492 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:04.234+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:04.492 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:04.234+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:04.493 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:04.234+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:04.493 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:04.303+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:04.493 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:04.303+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:04.493 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:04.303+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:04.493 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:04.304+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:04.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:04.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:04.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:04.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:04.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:04.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:04.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:04.516+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:05.486 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:05.199+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:05.486 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:05.199+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:05.486 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:05.199+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:05.486 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:05.200+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:05.487 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:05.337+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:05.487 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:05.337+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:05.487 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:05.337+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:05.487 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:05.338+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:05.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:05.487+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:05.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:05.487+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:05.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:05.487+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:05.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:05.488+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:06.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:06.446+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:06.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:06.446+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:06.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:06.446+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:06.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:06.448+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:06.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:06.341+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:06.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:06.341+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:06.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:06.341+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:06.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:06.342+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:06.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:06.218+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:06.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:06.218+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:06.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:06.218+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:06.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:06.219+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:07.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:07.409+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:07.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:07.409+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:07.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:07.409+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:07.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:07.410+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:07.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:07.292+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:07.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:07.292+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:07.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:07.292+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:07.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:07.293+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:07.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:07.185+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:07.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:07.185+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:07.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:07.185+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:07.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:07.186+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:08.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.480+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 msgr2=0x7fcfe4055420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:09:08.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.480+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 msgr2=0x7fcfe4055420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:09:08.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.480+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055420 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fcff0012ef0 tx=0x7fcfe80465e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:09:08.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.480+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055420 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fcff0012ef0 tx=0x7fcfe80465e0 comp rx=0 tx=0).stop 2024-04-27T04:09:08.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.480+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 msgr2=0x7fcfe4055420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:09:08.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.480+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055420 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:08.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.480+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:08.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.481+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:08.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.481+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcfe4054300 2024-04-27T04:09:08.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.481+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044da0 con 0x7fcff8101eb0 2024-04-27T04:09:08.481 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:08.279+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:08.482 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:08.279+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:08.482 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:08.279+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:08.482 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:08.279+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:08.482 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:08.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:08.482 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:08.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:08.482 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:08.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:08.482 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:08.218+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:08.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.481+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:09:08.483 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:08.460+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:08.483 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:08.460+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:08.483 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:08.460+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:08.483 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:08.460+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.481+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:09:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.481+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.481+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044ee0 con 0x7fcfe4054300 2024-04-27T04:09:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.482+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fcff000e900 tx=0x7fcff002b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:09:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003a340 con 0x7fcfe4054300 2024-04-27T04:09:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff002a860 con 0x7fcfe4054300 2024-04-27T04:09:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:08.484+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4054300 2024-04-27T04:09:08.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:09:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:09:08.481+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 863 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:09:09.455 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:09.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:09.455 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:09.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:09.455 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:09.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:09.455 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:09.306+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:09.456 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:09.197+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:09.456 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:09.197+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:09.456 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:09.197+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:09.456 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:09.198+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:09.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:09.456+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:09.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:09.456+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:09.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:09.456+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:09.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:09.457+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:10.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:10.286+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:10.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:10.286+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:10.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:10.286+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:10.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:10.287+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:10.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:10.449+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:10.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:10.449+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:10.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:10.449+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:10.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:10.450+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:10.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:10.211+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:10.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:10.211+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:10.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:10.211+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:10.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:10.212+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:11.472 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:11.309+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:11.472 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:11.309+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:11.472 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:11.309+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:11.473 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:11.310+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:11.473 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:11.199+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:11.473 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:11.199+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:11.473 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:11.199+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:11.473 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:11.199+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:11.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:11.473+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:11.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:11.473+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:11.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:11.473+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:11.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:11.473+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:12.484 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:12.302+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:12.485 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:12.302+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:12.485 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:12.302+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:12.485 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:12.302+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:12.485 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:12.222+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:12.485 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:12.222+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:12.485 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:12.222+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:12.485 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:12.222+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:12.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:12.485+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:12.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:12.485+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:12.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:12.485+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:12.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:12.486+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:13.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.481+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4054b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:09:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.481+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4054b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:09:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.481+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fcff000e900 tx=0x7fcff002b810 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:09:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.481+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fcff000e900 tx=0x7fcff002b810 comp rx=0 tx=0).stop 2024-04-27T04:09:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.481+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4054b00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:09:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.481+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.481+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.481+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.481+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcfe4054720 2024-04-27T04:09:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.481+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044da0 con 0x7fcff8101eb0 2024-04-27T04:09:13.481 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:13.307+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:13.482 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:13.307+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:13.482 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:13.307+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:13.482 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:13.308+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:13.482 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:13.212+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:13.482 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:13.212+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:13.482 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:13.212+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:13.482 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:13.213+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.482+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:09:13.483 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:13.437+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:13.483 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:13.437+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:13.483 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:13.437+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:13.483 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:13.438+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.483+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:09:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.483+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.483+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044ee0 con 0x7fcfe4054720 2024-04-27T04:09:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.483+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055040 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fcff0010830 tx=0x7fcff001c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:09:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.484+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003a340 con 0x7fcfe4054720 2024-04-27T04:09:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.484+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff002a860 con 0x7fcfe4054720 2024-04-27T04:09:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:13.485+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4054720 2024-04-27T04:09:13.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:09:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:09:13.481+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 924 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:09:14.613 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:14.419+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:14.613 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:14.419+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:14.614 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:14.419+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:14.614 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:14.420+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:14.614 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:14.180+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:14.614 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:14.180+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:14.614 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:14.180+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:14.614 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:14.181+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:14.614 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:14.344+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:14.614 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:14.344+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:14.615 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:14.344+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:14.615 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:14.345+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:14.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:14.901+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:09:14.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:14.901+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:09:15.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:15.174+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:15.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:15.174+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:15.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:15.174+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:15.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:15.174+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:15.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:15.398+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:15.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:15.398+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:15.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:15.398+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:15.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:15.398+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:15.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:15.391+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:15.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:15.391+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:15.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:15.391+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:15.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:15.392+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:16.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:16.164+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:16.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:16.164+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:16.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:16.164+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:16.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:16.165+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:16.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:16.355+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:16.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:16.355+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:16.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:16.355+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:16.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:16.356+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:16.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:16.440+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:16.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:16.440+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:16.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:16.440+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:16.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:16.441+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:17.142+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:17.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:17.142+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:17.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:17.142+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:17.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:17.143+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:17.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:17.382+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:17.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:17.382+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:17.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:17.382+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:17.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:17.382+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:17.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:17.458+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:17.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:17.458+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:17.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:17.458+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:17.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:17.459+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:18.115+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:18.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:18.115+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:18.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:18.115+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:18.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:18.116+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:18.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.481+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 msgr2=0x7fcfe4055040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:09:18.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.481+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 msgr2=0x7fcfe4055040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:09:18.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.482+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055040 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fcff0010830 tx=0x7fcff001c040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:09:18.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.482+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055040 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fcff0010830 tx=0x7fcff001c040 comp rx=0 tx=0).stop 2024-04-27T04:09:18.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 msgr2=0x7fcfe4055040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:09:18.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.482+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055040 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:18.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.482+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:18.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.482+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:18.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcfe4054300 2024-04-27T04:09:18.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044da0 con 0x7fcff8101eb0 2024-04-27T04:09:18.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.482+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:09:18.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.482+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:09:18.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.483+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:18.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.483+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044ee0 con 0x7fcfe4054300 2024-04-27T04:09:18.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.483+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4011230 tx=0x7fcff00103a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:09:18.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.483+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800cea0 con 0x7fcfe4054300 2024-04-27T04:09:18.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.484+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4054300 2024-04-27T04:09:18.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:18.485+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe804f160 con 0x7fcfe4054300 2024-04-27T04:09:18.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:09:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:09:18.482+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 986 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:09:18.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:18.431+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:18.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:18.431+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:18.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:18.431+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:18.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:18.432+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:18.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:18.440+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:18.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:18.440+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:18.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:18.440+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:18.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:18.441+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:19.390 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:19.113+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:19.390 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:19.113+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:19.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:19.113+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:19.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:19.114+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:19.665 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:19.391+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:19.665 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:19.391+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:19.665 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:19.391+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:19.665 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:19.392+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:19.665 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:19.409+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:19.666 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:19.409+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:19.666 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:19.409+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:19.666 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:19.410+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:20.415 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:20.369+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:20.415 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:20.369+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:20.415 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:20.369+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:20.415 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:20.370+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:20.415 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:20.143+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:20.415 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:20.143+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:20.416 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:20.143+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:20.416 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:20.143+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:20.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:20.415+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:20.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:20.415+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:20.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:20.415+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:20.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:20.416+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:21.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:21.388+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:21.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:21.388+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:21.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:21.388+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:21.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:21.390+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:21.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:21.395+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:21.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:21.395+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:21.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:21.395+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:21.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:21.396+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:21.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:21.168+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:21.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:21.168+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:21.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:21.168+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:21.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:21.168+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:22.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:22.341+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:22.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:22.341+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:22.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:22.341+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:22.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:22.342+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:22.352+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:22.352+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:22.352+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:22.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:22.354+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:22.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:22.136+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:22.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:22.136+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:22.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:22.136+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:22.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:22.136+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:23.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:23.349+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:23.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:23.349+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:23.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:23.349+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:23.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:23.350+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:23.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:23.312+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:23.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:23.312+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:23.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:23.312+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:23.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:23.313+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:23.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:23.113+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:23.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:23.113+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:23.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:23.113+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:23.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:23.113+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:23.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.482+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4054b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:09:23.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.482+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4054b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:09:23.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.482+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4011230 tx=0x7fcff00103a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:09:23.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.482+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4011230 tx=0x7fcff00103a0 comp rx=0 tx=0).stop 2024-04-27T04:09:23.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4054b00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:09:23.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.482+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:23.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.482+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:23.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.482+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:23.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcfe4054720 2024-04-27T04:09:23.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4044da0 con 0x7fcff8101eb0 2024-04-27T04:09:23.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.483+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:09:23.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.485+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:09:23.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.485+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:23.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.485+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044ee0 con 0x7fcfe4054720 2024-04-27T04:09:23.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.487+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055040 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804ddc0 tx=0x7fcfe8045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:09:23.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.488+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800cea0 con 0x7fcfe4054720 2024-04-27T04:09:23.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.488+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4054720 2024-04-27T04:09:23.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:23.488+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe804f160 con 0x7fcfe4054720 2024-04-27T04:09:23.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:09:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:09:23.482+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1046 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:09:24.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:24.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:24.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:24.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:24.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:24.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:24.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:24.075+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:24.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:24.302+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:24.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:24.302+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:24.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:24.302+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:24.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:24.303+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:24.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:24.322+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:24.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:24.322+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:24.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:24.322+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:24.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:24.323+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:25.337 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:25.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:25.337 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:25.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:25.337 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:25.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:25.337 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:25.082+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:25.338 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:25.325+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:25.338 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:25.325+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:25.338 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:25.325+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:25.338 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:25.325+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:25.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:25.338+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:25.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:25.338+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:25.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:25.338+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:25.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:25.339+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:25.757+0000 7fcfee7fc700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfdc005870 con 0x7fcfe4054720 2024-04-27T04:09:26.292 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:26.040+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:26.293 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:26.040+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:26.293 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:26.040+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:26.293 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:26.041+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:26.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:26.352+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:26.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:26.352+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:26.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:26.352+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:26.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:26.352+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:26.293+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:26.293+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:26.293+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:26.294+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:26.756+0000 7fcfee7fc700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfdc004790 con 0x7fcfe4054720 2024-04-27T04:09:27.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:26.994+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:27.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:26.994+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:27.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:26.994+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:27.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:26.995+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:27.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:27.335+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:27.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:27.335+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:27.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:27.335+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:27.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:27.336+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:27.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:27.325+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:27.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:27.325+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:27.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:27.325+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:27.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:27.326+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:27.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:27.756+0000 7fcfee7fc700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfdc004790 con 0x7fcfe4054720 2024-04-27T04:09:28.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:27.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:28.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:27.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:28.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:27.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:28.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:27.963+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:28.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.482+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 msgr2=0x7fcfe4055040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:09:28.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.482+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 msgr2=0x7fcfe4055040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:09:28.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.482+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055040 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804ddc0 tx=0x7fcfe8045000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:09:28.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.482+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055040 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804ddc0 tx=0x7fcfe8045000 comp rx=0 tx=0).stop 2024-04-27T04:09:28.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 msgr2=0x7fcfe4055040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:09:28.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.482+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054720 0x7fcfe4055040 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.482+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.482+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4046380 con 0x7fcfe4054300 2024-04-27T04:09:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcff8101eb0 2024-04-27T04:09:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.482+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:09:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.483+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:09:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.483+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.483+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4044da0 con 0x7fcfe4054300 2024-04-27T04:09:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.483+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fcff000cfe0 tx=0x7fcfe8046b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:09:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.484+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003a340 con 0x7fcfe4054300 2024-04-27T04:09:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.484+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0030f00 con 0x7fcfe4054300 2024-04-27T04:09:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:28.486+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4054300 2024-04-27T04:09:28.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:09:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:09:28.482+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1104 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:09:28.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:28.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:28.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:28.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:28.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:28.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:28.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:28.306+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:28.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:28.323+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:28.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:28.323+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:28.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:28.323+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:28.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:28.324+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:29.259 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:28.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:29.259 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:28.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:29.259 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:28.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:29.259 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:28.986+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:29.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:29.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:29.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:29.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:29.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:29.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:29.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:29.259+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:29.372+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:29.372+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:29.372+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:29.372+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:30.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:29.980+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:30.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:29.980+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:30.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:29.980+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:30.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:29.981+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:30.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:30.260+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:09:30.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:30.260+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:09:30.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:30.378+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:30.378+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:30.378+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:30.379+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:30.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:30.231+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:30.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:30.231+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:30.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:30.231+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:30.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:30.232+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:31.341 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:31.222+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:31.341 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:31.222+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:31.341 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:31.222+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:31.341 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:31.223+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:31.341 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:30.989+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:31.342 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:30.989+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:31.342 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:30.989+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:31.342 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:30.990+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:31.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:31.340+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:31.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:31.340+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:31.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:31.340+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:31.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:31.342+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:32.367 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:32.003+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:32.368 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:32.003+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:32.368 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:32.003+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:32.368 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:32.003+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:32.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:32.182+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:32.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:32.182+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:32.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:32.182+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:32.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:32.183+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:32.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:32.367+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:32.367+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:32.367+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:32.368+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:33.333 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:32.990+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:33.333 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:32.990+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:33.333 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:32.990+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:33.333 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:32.990+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:33.334 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:33.204+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:33.334 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:33.204+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:33.334 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:33.204+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:33.334 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:33.205+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:33.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.482+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4054b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:09:33.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.482+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4054b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:09:33.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.482+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fcff000cfe0 tx=0x7fcfe8046b70 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:09:33.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.482+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fcff000cfe0 tx=0x7fcfe8046b70 comp rx=0 tx=0).stop 2024-04-27T04:09:33.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.482+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4054b00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:09:33.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.482+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4054b00 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:33.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.483+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:33.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.483+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4055040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:33.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.483+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcfe4044cf0 2024-04-27T04:09:33.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.483+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40546e0 con 0x7fcff8101eb0 2024-04-27T04:09:33.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.483+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4055040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:09:33.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.484+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:09:33.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.484+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:33.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.484+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4054820 con 0x7fcfe4044cf0 2024-04-27T04:09:33.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.484+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4055040 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40102d0 tx=0x7fcff00303d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:09:33.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.484+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003a340 con 0x7fcfe4044cf0 2024-04-27T04:09:33.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.484+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff003aaf0 con 0x7fcfe4044cf0 2024-04-27T04:09:33.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:33.485+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4044cf0 2024-04-27T04:09:33.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:09:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:09:33.483+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1166 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:09:33.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:33.370+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:33.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:33.370+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:33.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:33.370+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:33.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:33.371+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:34.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:34.003+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:34.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:34.003+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:34.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:34.003+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:34.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:34.003+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:34.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:34.224+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:34.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:34.224+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:34.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:34.224+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:34.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:34.225+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:34.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:34.391+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:34.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:34.391+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:34.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:34.391+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:34.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:34.392+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:35.038+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:35.038+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:35.038+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:35.039+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:35.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:35.192+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:35.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:35.192+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:35.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:35.192+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:35.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:35.193+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:35.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:35.368+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:35.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:35.368+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:35.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:35.368+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:35.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:35.368+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:36.321 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:36.183+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:36.321 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:36.183+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:36.322 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:36.183+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:36.322 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:36.184+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:36.322 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:35.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:36.322 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:35.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:36.322 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:35.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:36.322 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:35.992+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:36.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:36.322+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:36.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:36.322+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:36.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:36.322+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:36.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:36.322+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:37.367 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:37.186+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:37.367 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:37.186+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:37.367 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:37.186+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:37.367 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:37.186+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:37.368 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:36.982+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:37.368 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:36.982+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:37.368 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:36.982+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:37.368 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:36.983+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:37.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:37.367+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:37.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:37.367+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:37.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:37.367+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:37.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:37.367+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:38.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:37.936+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:38.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:37.936+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:38.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:37.936+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:38.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:37.936+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:38.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.483+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4055040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:09:38.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.483+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4055040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:09:38.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.483+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4055040 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40102d0 tx=0x7fcff00303d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:09:38.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.483+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4055040 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40102d0 tx=0x7fcff00303d0 comp rx=0 tx=0).stop 2024-04-27T04:09:38.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.483+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4055040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:09:38.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.483+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4055040 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:38.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.483+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:38.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.483+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:38.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.483+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcfe4054300 2024-04-27T04:09:38.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.483+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4054c20 con 0x7fcff8101eb0 2024-04-27T04:09:38.485 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:38.212+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:38.485 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:38.212+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:38.485 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:38.212+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:38.485 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:38.212+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:38.485 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:38.378+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:38.485 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:38.378+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:38.485 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:38.378+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:38.485 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:38.378+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:38.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.484+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:09:38.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.484+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:09:38.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.484+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.484+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4054d60 con 0x7fcfe4054300 2024-04-27T04:09:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.484+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fcff0023950 tx=0x7fcff0030810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:09:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.486+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00132f0 con 0x7fcfe4054300 2024-04-27T04:09:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.487+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0009e80 con 0x7fcfe4054300 2024-04-27T04:09:38.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:38.488+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0010d60 con 0x7fcfe4054300 2024-04-27T04:09:38.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:09:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:09:38.483+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1229 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:09:39.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:39.165+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:39.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:39.165+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:39.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:39.165+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:39.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:39.165+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:39.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:38.980+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:39.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:38.980+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:39.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:38.980+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:39.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:38.981+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:39.916 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:39.429+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:39.917 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:39.429+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:39.917 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:39.429+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:39.917 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:39.429+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:40.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:40.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:40.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:40.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:40.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:40.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:40.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:40.164+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:40.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:40.019+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:40.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:40.019+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:40.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:40.019+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:40.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:40.019+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:40.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:40.424+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:40.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:40.424+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:40.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:40.424+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:40.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:40.425+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:41.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:41.120+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:41.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:41.120+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:41.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:41.120+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:41.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:41.121+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:41.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:40.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:41.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:40.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:41.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:40.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:41.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:40.998+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:41.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:41.458+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:41.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:41.458+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:41.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:41.458+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:41.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:41.459+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:42.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:42.038+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:42.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:42.038+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:42.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:42.038+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:42.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:42.039+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:42.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:42.119+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:42.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:42.119+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:42.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:42.119+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:42.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:42.121+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:42.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:42.445+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:42.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:42.445+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:42.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:42.445+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:42.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:42.447+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:43.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:43.030+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:43.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:43.030+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:43.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:43.030+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:43.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:43.031+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:43.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:43.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:43.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:43.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:43.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:43.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:43.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:43.165+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:43.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.484+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:09:43.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.484+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:09:43.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.485+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fcff0023950 tx=0x7fcff0030810 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:09:43.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.485+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fcff0023950 tx=0x7fcff0030810 comp rx=0 tx=0).stop 2024-04-27T04:09:43.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.485+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:09:43.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.485+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:43.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.485+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:43.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.485+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:43.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.485+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f3e0 con 0x7fcfe4044cf0 2024-04-27T04:09:43.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.485+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4055160 con 0x7fcff8101eb0 2024-04-27T04:09:43.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.485+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:09:43.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.486+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:09:43.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.486+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.486+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40552a0 con 0x7fcfe4044cf0 2024-04-27T04:09:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.486+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fcfe805b5a0 tx=0x7fcff0030f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:09:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.487+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4044cf0 2024-04-27T04:09:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.487+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4044cf0 2024-04-27T04:09:43.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:43.488+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe80363c0 con 0x7fcfe4044cf0 2024-04-27T04:09:43.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:09:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:09:43.485+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1290 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:09:43.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:43.439+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:43.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:43.439+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:43.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:43.439+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:43.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:43.439+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:44.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:44.038+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:44.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:44.038+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:44.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:44.038+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:44.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:44.039+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:44.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:44.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:44.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:44.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:44.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:44.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:44.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:44.131+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:44.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:44.480+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:44.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:44.480+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:44.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:44.480+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:44.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:44.480+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:45.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:45.140+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:45.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:45.140+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:45.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:45.140+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:45.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:45.141+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:45.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:45.039+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:45.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:45.039+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:45.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:45.039+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:45.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:45.040+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:45.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:45.621+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:09:45.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:45.621+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:09:45.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:45.524+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:45.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:45.524+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:45.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:45.524+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:45.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:45.525+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:46.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:46.109+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:46.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:46.109+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:46.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:46.109+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:46.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:46.110+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:46.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:46.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:46.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:46.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:46.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:46.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:46.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:46.018+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:46.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:46.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:46.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:46.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:46.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:46.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:46.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:46.512+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:47.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:47.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:47.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:47.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:47.082+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:47.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:47.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:47.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:47.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:47.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:47.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:47.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:47.023+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:47.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:47.505+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:47.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:47.505+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:47.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:47.505+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:47.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:47.506+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:48.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:48.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:48.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:48.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:48.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:48.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:48.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:48.112+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:48.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:48.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:48.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:48.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:48.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:48.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:48.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:48.033+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.485+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:09:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.485+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:09:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.485+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fcfe805b5a0 tx=0x7fcff0030f60 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:09:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.485+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fcfe805b5a0 tx=0x7fcff0030f60 comp rx=0 tx=0).stop 2024-04-27T04:09:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.485+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:09:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.485+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.485+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.485+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.485+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f310 con 0x7fcfe4054300 2024-04-27T04:09:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.485+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f450 con 0x7fcff8101eb0 2024-04-27T04:09:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.486+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:09:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.486+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:09:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.486+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.486+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4055160 con 0x7fcfe4054300 2024-04-27T04:09:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.487+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fcff004a5c0 tx=0x7fcfe8027000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:09:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.488+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00132f0 con 0x7fcfe4054300 2024-04-27T04:09:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.488+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0009e80 con 0x7fcfe4054300 2024-04-27T04:09:48.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:48.489+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff003a520 con 0x7fcfe4054300 2024-04-27T04:09:48.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:09:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:09:48.485+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1352 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:09:48.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:48.539+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:48.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:48.539+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:48.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:48.539+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:48.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:48.540+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:49.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:49.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:49.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:49.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:49.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:49.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:49.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:49.112+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:49.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:49.065+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:49.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:49.065+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:49.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:49.065+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:49.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:49.066+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:49.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:49.495+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:49.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:49.495+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:49.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:49.495+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:49.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:49.496+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:50.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:50.158+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:50.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:50.158+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:50.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:50.158+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:50.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:50.158+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:50.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:50.099+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:50.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:50.099+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:50.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:50.099+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:50.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:50.100+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:50.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:50.497+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:50.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:50.497+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:50.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:50.497+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:50.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:50.498+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:51.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:51.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:51.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:51.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:51.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:51.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:51.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:51.112+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:51.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:51.101+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:51.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:51.101+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:51.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:51.101+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:51.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:51.102+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:51.765 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:51.472+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:51.765 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:51.472+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:51.765 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:51.472+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:51.765 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:51.473+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:52.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:52.147+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:52.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:52.147+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:52.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:52.147+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:52.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:52.148+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:52.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:52.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:52.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:52.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:52.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:52.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:52.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:52.148+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:52.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:52.496+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:52.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:52.496+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:52.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:52.496+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:52.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:52.497+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:53.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:53.107+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:53.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:53.107+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:53.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:53.107+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:53.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:53.108+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:53.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:53.165+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:53.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:53.165+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:53.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:53.165+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:53.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:53.166+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:53.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.485+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:09:53.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.486+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:09:53.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.486+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fcff004a5c0 tx=0x7fcfe8027000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:09:53.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.486+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fcff004a5c0 tx=0x7fcfe8027000 comp rx=0 tx=0).stop 2024-04-27T04:09:53.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.486+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:09:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.486+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.486+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.486+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.486+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f310 con 0x7fcfe4044cf0 2024-04-27T04:09:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.486+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f450 con 0x7fcff8101eb0 2024-04-27T04:09:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.486+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:09:53.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.487+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:09:53.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.487+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:53.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.487+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4055160 con 0x7fcfe4044cf0 2024-04-27T04:09:53.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.487+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4017c70 tx=0x7fcff001c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:09:53.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.487+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00132f0 con 0x7fcfe4044cf0 2024-04-27T04:09:53.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.487+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff003a8e0 con 0x7fcfe4044cf0 2024-04-27T04:09:53.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:53.489+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff003a520 con 0x7fcfe4044cf0 2024-04-27T04:09:53.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:09:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:09:53.486+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1414 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:09:53.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:53.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:53.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:53.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:53.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:53.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:53.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:53.517+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:54.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:54.102+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:54.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:54.102+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:54.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:54.102+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:54.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:54.103+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:54.175+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:54.175+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:54.175+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:54.176+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:54.836 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:54.514+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:54.836 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:54.514+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:54.837 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:54.514+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:54.837 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:54.514+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:55.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:55.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:55.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:55.087+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:55.535 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:55.187+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:55.536 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:55.187+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:55.536 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:55.187+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:55.536 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:55.188+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:55.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:55.536+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:55.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:55.536+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:55.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:55.536+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:55.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:55.537+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:56.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:56.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:56.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:56.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:56.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:56.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:56.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:56.069+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:56.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:56.210+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:56.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:56.210+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:56.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:56.210+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:56.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:56.211+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:56.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:56.554+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:56.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:56.554+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:56.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:56.554+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:56.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:56.555+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:57.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:57.091+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:57.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:57.091+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:57.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:57.091+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:57.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:57.092+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:57.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:57.237+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:57.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:57.237+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:57.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:57.237+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:57.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:57.238+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:57.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:57.604+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:57.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:57.604+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:57.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:57.604+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:57.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:57.605+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:58.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:58.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:58.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:58.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:58.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:58.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:58.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:58.077+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:58.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:58.193+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:58.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:58.193+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:58.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:58.193+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:58.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:58.194+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:58.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.486+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:09:58.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.486+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:09:58.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.486+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4017c70 tx=0x7fcff001c070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:09:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.486+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4017c70 tx=0x7fcff001c070 comp rx=0 tx=0).stop 2024-04-27T04:09:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.486+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:09:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.486+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.486+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.486+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:09:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.486+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f310 con 0x7fcfe4054300 2024-04-27T04:09:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.486+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f450 con 0x7fcff8101eb0 2024-04-27T04:09:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.486+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:09:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.487+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:09:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.487+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:09:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.487+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4055160 con 0x7fcfe4054300 2024-04-27T04:09:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.488+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40137f0 tx=0x7fcff0041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:09:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.488+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4054300 2024-04-27T04:09:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.488+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4054300 2024-04-27T04:09:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:09:58.489+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8008d70 con 0x7fcfe4054300 2024-04-27T04:09:58.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:09:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:09:58.486+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1477 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:09:58.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:58.562+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:58.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:58.562+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:58.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:58.562+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:58.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:58.564+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:09:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:59.199+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:59.199+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:59.199+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:09:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:09:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:09:59.200+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:59.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:59.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:59.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:09:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:09:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:09:59.062+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:09:59.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:59.544+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:59.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:59.544+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:59.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:59.544+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:09:59.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:09:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:09:59.546+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:00.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:00.172+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:00.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:00.172+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:00.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:00.172+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:00.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:00.173+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:00.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:00.018+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:00.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:00.018+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:00.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:00.018+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:00.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:00.019+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:00.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:00.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:00.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:00.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:00.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:00.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:00.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:00.512+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:00.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:00.981+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:10:00.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:00.981+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:10:01.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:01.004+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:01.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:01.004+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:01.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:01.004+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:01.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:01.004+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:01.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:01.197+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:01.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:01.197+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:01.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:01.197+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:01.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:01.198+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:01.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:01.510+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:01.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:01.510+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:01.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:01.510+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:01.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:01.511+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:02.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:02.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:02.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:02.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:02.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:02.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:02.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:02.215+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:02.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:02.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:02.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:02.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:02.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:02.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:02.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:02.048+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:02.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:02.760+0000 7fcfee7fc700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfdc004790 con 0x7fcfe4054300 2024-04-27T04:10:02.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:02.539+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:02.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:02.539+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:02.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:02.539+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:02.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:02.539+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:03.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:03.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:03.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:03.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:03.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:03.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:03.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:03.189+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:03.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:03.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:03.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:03.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:03.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:03.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:03.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:03.036+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:03.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.487+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:10:03.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.487+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:10:03.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.487+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40137f0 tx=0x7fcff0041040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:10:03.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.487+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40137f0 tx=0x7fcff0041040 comp rx=0 tx=0).stop 2024-04-27T04:10:03.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.487+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:10:03.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.487+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:03.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.487+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:03.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.487+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:03.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.487+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f4b0 con 0x7fcfe4044cf0 2024-04-27T04:10:03.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.487+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4055160 con 0x7fcff8101eb0 2024-04-27T04:10:03.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.488+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:10:03.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.488+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:10:03.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.488+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:03.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.488+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40552a0 con 0x7fcfe4044cf0 2024-04-27T04:10:03.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.489+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fcff0013960 tx=0x7fcff003fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:10:03.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.489+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4044cf0 2024-04-27T04:10:03.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.490+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037900 con 0x7fcfe4044cf0 2024-04-27T04:10:03.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:03.490+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8031b70 con 0x7fcfe4044cf0 2024-04-27T04:10:03.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:10:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:10:03.487+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1538 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:10:03.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:03.502+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:03.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:03.502+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:03.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:03.502+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:03.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:03.503+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:04.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:04.167+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:04.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:04.167+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:04.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:04.167+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:04.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:04.168+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:04.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:04.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:04.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:04.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:04.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:04.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:04.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:04.060+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:04.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:04.481+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:04.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:04.481+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:04.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:04.481+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:04.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:04.482+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:05.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:05.053+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:05.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:05.053+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:05.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:05.053+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:05.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:05.054+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:05.132+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:05.132+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:05.132+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:05.132+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:05.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:05.484+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:05.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:05.484+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:05.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:05.484+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:05.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:05.484+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:06.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:06.177+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:06.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:06.177+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:06.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:06.177+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:06.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:06.178+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:06.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:06.027+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:06.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:06.027+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:06.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:06.027+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:06.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:06.028+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:06.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:06.497+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:06.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:06.497+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:06.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:06.497+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:06.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:06.498+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:07.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:07.159+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:07.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:07.159+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:07.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:07.159+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:07.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:07.160+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:07.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:07.037+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:07.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:07.037+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:07.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:07.037+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:07.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:07.038+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:07.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:07.483+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:07.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:07.483+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:07.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:07.483+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:07.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:07.484+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:08.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:08.200+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:08.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:08.200+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:08.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:08.200+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:08.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:08.201+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:08.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:08.055+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:08.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:08.055+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:08.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:08.055+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:08.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:08.056+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.487+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:10:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.487+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:10:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.487+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fcff0013960 tx=0x7fcff003fac0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:10:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.487+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fcff0013960 tx=0x7fcff003fac0 comp rx=0 tx=0).stop 2024-04-27T04:10:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.487+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:10:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.487+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.488+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.488+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.488+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f4b0 con 0x7fcfe4054300 2024-04-27T04:10:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.488+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4055160 con 0x7fcff8101eb0 2024-04-27T04:10:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.488+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:10:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.489+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:10:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.489+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.489+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40552a0 con 0x7fcfe4054300 2024-04-27T04:10:08.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.489+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fcff003a7d0 tx=0x7fcfe8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:10:08.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.490+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00132f0 con 0x7fcfe4054300 2024-04-27T04:10:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.492+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0037580 con 0x7fcfe4054300 2024-04-27T04:10:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:08.492+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff003a520 con 0x7fcfe4054300 2024-04-27T04:10:08.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:10:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:10:08.487+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1600 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:10:08.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:08.487+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:08.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:08.487+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:08.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:08.487+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:08.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:08.488+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:09.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:09.171+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:09.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:09.171+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:09.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:09.171+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:09.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:09.172+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:09.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:09.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:09.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:09.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:09.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:09.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:09.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:09.064+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:09.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:09.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:09.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:09.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:09.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:09.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:09.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:09.448+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:10.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:10.151+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:10.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:10.151+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:10.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:10.151+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:10.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:10.152+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:10.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:10.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:10.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:10.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:10.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:10.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:10.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:10.071+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:10.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:10.482+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:10.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:10.482+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:10.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:10.482+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:10.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:10.483+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:11.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:11.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:11.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:11.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:11.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:11.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:11.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:11.118+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:11.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:11.098+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:11.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:11.098+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:11.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:11.098+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:11.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:11.099+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:11.761+0000 7fcfee7fc700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfdc004790 con 0x7fcfe4054300 2024-04-27T04:10:11.761 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:11.453+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:11.761 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:11.453+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:11.761 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:11.453+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:11.761 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:11.454+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:12.411 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:12.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:12.411 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:12.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:12.411 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:12.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:12.411 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:12.077+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:12.411 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:12.165+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:12.411 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:12.165+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:12.411 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:12.165+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:12.412 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:12.166+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:12.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:12.411+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:12.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:12.411+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:12.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:12.411+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:12.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:12.412+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:12.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:12.762+0000 7fcfee7fc700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfdc0059d0 con 0x7fcfe4054300 2024-04-27T04:10:13.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:13.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:13.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:13.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:13.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:13.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:13.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:13.063+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:13.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:13.185+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:13.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:13.185+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:13.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:13.185+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:13.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:13.185+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:13.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.488+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:10:13.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.488+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:10:13.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.488+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fcff003a7d0 tx=0x7fcfe8019000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:10:13.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.488+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fcff003a7d0 tx=0x7fcfe8019000 comp rx=0 tx=0).stop 2024-04-27T04:10:13.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.488+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:10:13.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.488+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:13.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.488+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:13.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.488+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:13.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.488+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f4b0 con 0x7fcfe4044cf0 2024-04-27T04:10:13.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.488+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4055160 con 0x7fcff8101eb0 2024-04-27T04:10:13.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.488+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:10:13.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.489+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:10:13.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.489+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:13.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.489+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40552a0 con 0x7fcfe4044cf0 2024-04-27T04:10:13.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.490+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400eea0 tx=0x7fcfe8014900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:10:13.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.491+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0037340 con 0x7fcfe4044cf0 2024-04-27T04:10:13.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.491+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0037170 con 0x7fcfe4044cf0 2024-04-27T04:10:13.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:13.492+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0013110 con 0x7fcfe4044cf0 2024-04-27T04:10:13.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:10:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:10:13.488+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1662 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:10:13.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:13.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:13.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:13.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:13.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:13.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:13.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:13.449+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:14.404 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:14.093+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:14.404 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:14.093+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:14.404 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:14.093+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:14.404 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:14.094+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:14.404 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:14.160+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:14.404 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:14.160+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:14.404 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:14.160+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:14.404 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:14.161+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:14.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:14.404+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:14.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:14.404+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:14.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:14.404+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:14.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:14.406+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:15.366 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:15.128+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:15.366 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:15.128+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:15.366 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:15.128+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:15.366 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:15.129+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:15.367 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:15.108+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:15.367 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:15.108+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:15.367 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:15.108+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:15.367 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:15.109+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:15.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:15.367+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:15.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:15.367+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:15.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:15.367+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:15.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:15.368+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:16.341+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:10:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:16.341+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:10:16.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:16.096+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:16.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:16.096+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:16.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:16.096+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:16.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:16.097+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:16.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:16.123+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:16.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:16.123+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:16.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:16.123+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:16.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:16.124+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:16.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:16.344+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:16.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:16.344+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:16.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:16.344+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:16.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:16.344+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:17.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:17.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:17.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:17.131+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:17.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:17.357+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:17.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:17.357+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:17.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:17.357+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:17.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:17.358+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:17.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:17.148+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:17.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:17.148+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:17.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:17.148+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:17.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:17.149+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:18.108+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:18.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:18.108+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:18.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:18.108+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:18.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:18.109+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:18.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:18.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:18.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:18.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:18.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:18.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:18.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:18.148+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:18.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:18.327+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:18.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:18.327+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:18.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:18.327+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:18.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:18.328+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:18.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.488+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:10:18.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.488+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:10:18.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.488+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400eea0 tx=0x7fcfe8014900 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:10:18.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.488+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400eea0 tx=0x7fcfe8014900 comp rx=0 tx=0).stop 2024-04-27T04:10:18.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.488+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:10:18.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.488+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:18.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.488+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:18.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.489+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:18.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.489+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f4b0 con 0x7fcfe4054300 2024-04-27T04:10:18.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.489+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4055160 con 0x7fcff8101eb0 2024-04-27T04:10:18.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.489+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:10:18.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.489+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:10:18.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.489+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:18.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.490+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40552a0 con 0x7fcfe4054300 2024-04-27T04:10:18.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.490+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fcff0013f30 tx=0x7fcff001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:10:18.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.491+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0037340 con 0x7fcfe4054300 2024-04-27T04:10:18.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.492+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff00199d0 con 0x7fcfe4054300 2024-04-27T04:10:18.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:18.493+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4054300 2024-04-27T04:10:18.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:10:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:10:18.489+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1725 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:10:19.365 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:19.171+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:19.365 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:19.171+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:19.365 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:19.171+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:19.365 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:19.172+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:19.365 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:19.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:19.365 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:19.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:19.365 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:19.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:19.365 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:19.090+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:19.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:19.365+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:19.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:19.365+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:19.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:19.365+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:19.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:19.366+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:20.369 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:20.097+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:20.370 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:20.097+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:20.370 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:20.097+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:20.370 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:20.097+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:20.370 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:20.135+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:20.370 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:20.135+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:20.370 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:20.135+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:20.370 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:20.136+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:20.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:20.410+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:20.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:20.410+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:20.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:20.410+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:20.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:20.411+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:21.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:21.108+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:21.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:21.108+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:21.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:21.108+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:21.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:21.109+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:21.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:21.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:21.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:21.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:21.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:21.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:21.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:21.049+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:21.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:21.459+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:21.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:21.459+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:21.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:21.459+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:21.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:21.460+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:22.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:22.081+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:22.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:22.081+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:22.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:22.081+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:22.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:22.082+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:22.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:22.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:22.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:22.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:22.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:22.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:22.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:22.053+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:22.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:22.491+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:22.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:22.491+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:22.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:22.491+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:22.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:22.492+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:23.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:23.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:23.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:23.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:23.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:23.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:23.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:23.048+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:23.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:23.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:23.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:23.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:23.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:23.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:23.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:23.082+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:23.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.490+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:10:23.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.490+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:10:23.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.490+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fcff0013f30 tx=0x7fcff001b000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:10:23.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.490+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fcff0013f30 tx=0x7fcff001b000 comp rx=0 tx=0).stop 2024-04-27T04:10:23.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.490+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:10:23.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.490+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:23.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.490+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:23.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.490+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:23.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.490+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051c20 con 0x7fcfe4044cf0 2024-04-27T04:10:23.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.490+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f4b0 con 0x7fcff8101eb0 2024-04-27T04:10:23.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.490+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:10:23.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.491+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:10:23.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.491+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:23.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.491+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4055160 con 0x7fcfe4044cf0 2024-04-27T04:10:23.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.491+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4010540 tx=0x7fcff000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:10:23.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.492+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0037340 con 0x7fcfe4044cf0 2024-04-27T04:10:23.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.492+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff00199d0 con 0x7fcfe4044cf0 2024-04-27T04:10:23.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:23.492+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4044cf0 2024-04-27T04:10:23.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:10:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:10:23.490+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1786 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:10:23.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:23.451+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:23.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:23.451+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:23.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:23.451+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:23.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:23.452+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:24.417 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:24.003+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:24.417 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:24.003+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:24.417 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:24.003+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:24.418 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:24.004+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:24.418 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:24.121+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:24.418 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:24.121+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:24.418 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:24.121+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:24.418 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:24.122+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:24.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:24.417+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:24.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:24.417+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:24.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:24.417+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:24.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:24.419+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:25.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:24.997+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:25.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:24.997+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:25.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:24.997+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:25.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:24.998+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:25.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:25.121+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:25.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:25.121+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:25.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:25.121+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:25.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:25.122+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:25.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:25.392+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:25.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:25.392+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:25.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:25.392+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:25.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:25.394+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:26.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:26.033+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:26.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:26.033+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:26.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:26.033+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:26.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:26.034+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:26.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:26.083+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:26.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:26.084+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:26.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:26.084+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:26.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:26.084+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:26.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:26.423+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:26.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:26.423+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:26.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:26.423+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:26.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:26.424+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:27.385 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:27.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:27.385 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:27.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:27.385 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:27.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:27.385 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:27.041+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:27.385 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:27.038+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:27.385 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:27.039+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:27.385 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:27.039+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:27.386 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:27.039+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:27.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:27.385+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:27.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:27.385+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:27.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:27.385+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:27.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:27.386+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:28.386 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:28.084+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:28.387 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:28.084+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:28.387 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:28.084+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:28.387 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:28.086+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:28.387 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:28.082+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:28.387 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:28.082+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:28.387 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:28.082+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:28.387 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:28.082+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.490+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:10:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.490+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:10:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.490+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4010540 tx=0x7fcff000ee50 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:10:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.490+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4010540 tx=0x7fcff000ee50 comp rx=0 tx=0).stop 2024-04-27T04:10:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.490+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:10:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.490+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.490+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.490+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.490+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051c20 con 0x7fcfe4054300 2024-04-27T04:10:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.491+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f4b0 con 0x7fcff8101eb0 2024-04-27T04:10:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.491+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:10:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.491+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:10:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.491+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.491+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4055160 con 0x7fcfe4054300 2024-04-27T04:10:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.491+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400ebe0 tx=0x7fcff0011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:10:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.492+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4054300 2024-04-27T04:10:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.492+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4054300 2024-04-27T04:10:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:28.493+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe805d420 con 0x7fcfe4054300 2024-04-27T04:10:28.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:10:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:10:28.490+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1849 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:10:28.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:28.387+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:28.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:28.387+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:28.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:28.387+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:28.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:28.388+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:29.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:29.053+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:29.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:29.053+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:29.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:29.053+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:29.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:29.053+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:29.379 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:29.132+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:29.379 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:29.132+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:29.379 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:29.132+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:29.379 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:29.132+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:29.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:29.379+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:29.379+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:29.379+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:29.381+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:30.338 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:30.075+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:30.338 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:30.075+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:30.339 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:30.075+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:30.339 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:30.076+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:30.339 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:30.102+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:30.339 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:30.102+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:30.339 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:30.102+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:30.339 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:30.102+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:30.339+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:30.339+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:30.339+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:30.340+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:31.369 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:31.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:31.369 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:31.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:31.369 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:31.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:31.369 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:31.067+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:31.370 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:31.044+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:31.370 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:31.044+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:31.370 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:31.044+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:31.370 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:31.044+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:31.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:31.370+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:31.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:31.370+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:31.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:31.370+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:31.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:31.371+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:31.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:31.701+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:10:31.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:31.701+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:10:32.380 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:32.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:32.381 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:32.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:32.381 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:32.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:32.381 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:32.063+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:32.381 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:32.094+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:32.381 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:32.094+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:32.381 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:32.094+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:32.381 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:32.095+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:32.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:32.381+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:32.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:32.381+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:32.381+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:32.383+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:33.378 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:33.033+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:33.378 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:33.033+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:33.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:33.033+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:33.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:33.033+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:33.379 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:33.122+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:33.379 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:33.122+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:33.379 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:33.122+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:33.379 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:33.122+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:33.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.491+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:10:33.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.491+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:10:33.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.491+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400ebe0 tx=0x7fcff0011bb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:10:33.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.491+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400ebe0 tx=0x7fcff0011bb0 comp rx=0 tx=0).stop 2024-04-27T04:10:33.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.491+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40546e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:10:33.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.491+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40546e0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:33.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.491+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:33.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.491+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:33.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.491+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051c20 con 0x7fcfe4044cf0 2024-04-27T04:10:33.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.491+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f4b0 con 0x7fcff8101eb0 2024-04-27T04:10:33.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.491+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:10:33.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.492+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:10:33.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.492+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:33.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.492+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4055160 con 0x7fcfe4044cf0 2024-04-27T04:10:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.493+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4018490 tx=0x7fcfe800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:10:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.493+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0037340 con 0x7fcfe4044cf0 2024-04-27T04:10:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.494+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0033000 con 0x7fcfe4044cf0 2024-04-27T04:10:33.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:33.495+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4044cf0 2024-04-27T04:10:33.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:10:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:10:33.491+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1912 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:10:33.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:33.379+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:33.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:33.379+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:33.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:33.379+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:33.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:33.380+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:34.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:34.159+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:34.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:34.159+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:34.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:34.159+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:34.381 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:34.160+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:34.381 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:34.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:34.381 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:34.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:34.381 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:34.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:34.381 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:34.068+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:34.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:34.381+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:34.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:34.381+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:34.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:34.381+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:34.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:34.382+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:35.417 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:35.080+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:35.418 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:35.080+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:35.418 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:35.080+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:35.418 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:35.081+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:35.418 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:35.418+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:35.418 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:35.418+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:35.418 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:35.418+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:35.419 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:35.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:35.419 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:35.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:35.419 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:35.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:35.419 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:35.203+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:35.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:35.419+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:36.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:36.090+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:36.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:36.090+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:36.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:36.090+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:36.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:36.092+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:36.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:36.195+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:36.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:36.195+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:36.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:36.195+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:36.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:36.196+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:36.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:36.438+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:36.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:36.438+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:36.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:36.438+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:36.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:36.439+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:37.391 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:37.120+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:37.391 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:37.120+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:37.391 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:37.120+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:37.391 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:37.121+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:37.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:37.210+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:37.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:37.210+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:37.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:37.210+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:37.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:37.211+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:37.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:37.392+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:37.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:37.392+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:37.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:37.392+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:37.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:37.393+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:38.389 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:38.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:38.389 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:38.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:38.389 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:38.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:38.389 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:38.095+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:38.389 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:38.201+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:38.389 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:38.201+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:38.389 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:38.201+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:38.389 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:38.202+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:38.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.492+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:10:38.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.492+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:10:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.492+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4018490 tx=0x7fcfe800bda0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:10:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.492+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4018490 tx=0x7fcfe800bda0 comp rx=0 tx=0).stop 2024-04-27T04:10:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.492+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:10:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.492+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.492+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.492+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.492+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051c20 con 0x7fcfe4054300 2024-04-27T04:10:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.492+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f4b0 con 0x7fcff8101eb0 2024-04-27T04:10:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.492+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:10:38.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.493+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40546e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:10:38.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.493+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40546e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:38.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.493+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4055160 con 0x7fcfe4054300 2024-04-27T04:10:38.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.494+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009790 tx=0x7fcff0003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:10:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.494+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0019c80 con 0x7fcfe4054300 2024-04-27T04:10:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.495+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff002a240 con 0x7fcfe4054300 2024-04-27T04:10:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:38.495+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4054300 2024-04-27T04:10:38.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:10:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:10:38.492+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 1975 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:10:38.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:38.389+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:38.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:38.389+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:38.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:38.389+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:38.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:38.390+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:39.383 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:39.166+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:39.384 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:39.166+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:39.384 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:39.166+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:39.384 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:39.167+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:39.384 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:39.077+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:39.384 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:39.077+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:39.384 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:39.077+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:39.384 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:39.077+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:39.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:39.384+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:39.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:39.384+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:39.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:39.384+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:39.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:39.385+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:40.359 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:40.190+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:40.359 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:40.190+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:40.359 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:40.190+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:40.359 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:40.191+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:40.360 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:40.075+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:40.360 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:40.075+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:40.360 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:40.075+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:40.360 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:40.075+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:40.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:40.360+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:40.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:40.360+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:40.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:40.360+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:40.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:40.361+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:41.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:41.030+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:41.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:41.030+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:41.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:41.030+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:41.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:41.030+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:41.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:41.229+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:41.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:41.229+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:41.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:41.229+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:41.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:41.229+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:41.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:41.398+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:41.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:41.398+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:41.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:41.398+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:41.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:41.400+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:42.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:42.206+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:42.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:42.206+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:42.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:42.206+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:42.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:42.207+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:42.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:42.030+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:42.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:42.030+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:42.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:42.030+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:42.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:42.031+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:42.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:42.431+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:42.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:42.431+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:42.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:42.431+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:42.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:42.432+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:43.406 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:43.037+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:43.406 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:43.037+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:43.406 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:43.037+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:43.406 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:43.037+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:43.406 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:43.157+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:43.406 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:43.157+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:43.406 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:43.157+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:43.407 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:43.158+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.492+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:10:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.492+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:10:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.492+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009790 tx=0x7fcff0003e00 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:10:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.492+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009790 tx=0x7fcff0003e00 comp rx=0 tx=0).stop 2024-04-27T04:10:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.492+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:10:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.492+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.492+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:43.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.492+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:43.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.492+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051c20 con 0x7fcfe4044cf0 2024-04-27T04:10:43.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.492+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f4b0 con 0x7fcff8101eb0 2024-04-27T04:10:43.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.493+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:10:43.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.493+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40546e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:10:43.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.494+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40546e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:43.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.494+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4055160 con 0x7fcfe4044cf0 2024-04-27T04:10:43.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.494+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009790 tx=0x7fcff000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:10:43.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.494+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4044cf0 2024-04-27T04:10:43.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.495+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4044cf0 2024-04-27T04:10:43.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:43.496+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013510 con 0x7fcfe4044cf0 2024-04-27T04:10:43.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:10:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:10:43.493+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 2036 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:10:43.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:43.406+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:43.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:43.406+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:43.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:43.406+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:43.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:43.407+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:44.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:44.027+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:44.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:44.027+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:44.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:44.027+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:44.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:44.027+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:44.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:44.130+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:44.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:44.130+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:44.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:44.131+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:44.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:44.131+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:44.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:44.392+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:44.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:44.392+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:44.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:44.392+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:44.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:44.393+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:45.398 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:45.010+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:45.398 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:45.010+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:45.398 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:45.010+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:45.398 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:45.011+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:45.399 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:45.090+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:45.399 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:45.090+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:45.399 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:45.090+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:45.399 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:45.091+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:45.648 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:45.399+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:45.648 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:45.399+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:45.648 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:45.399+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:45.648 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:45.399+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:46.398 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:46.043+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:46.398 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:46.043+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:46.398 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:46.043+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:46.398 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:46.043+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:46.398 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:45.976+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:46.399 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:45.976+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:46.399 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:45.976+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:46.399 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:45.977+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:46.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:46.399+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:46.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:46.399+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:46.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:46.399+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:46.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:46.399+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:47.061+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:10:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:47.061+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:10:47.367 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:47.030+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:47.367 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:47.030+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:47.368 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:47.030+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:47.368 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:47.031+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:47.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:46.987+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:47.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:46.987+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:47.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:46.987+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:47.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:46.987+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:47.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:47.368+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:47.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:47.368+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:47.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:47.368+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:47.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:47.369+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:48.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:47.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:48.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:47.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:48.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:47.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:48.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:47.998+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:48.352 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:48.013+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:48.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:48.013+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:48.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:48.013+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:48.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:48.014+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.493+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:10:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.493+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:10:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.493+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009790 tx=0x7fcff000ee50 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:10:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.493+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009790 tx=0x7fcff000ee50 comp rx=0 tx=0).stop 2024-04-27T04:10:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.493+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4054c20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:10:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.493+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4054c20 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.493+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.493+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:48.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.493+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051c20 con 0x7fcfe4054300 2024-04-27T04:10:48.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.493+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f4b0 con 0x7fcff8101eb0 2024-04-27T04:10:48.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.494+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:10:48.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.495+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40546e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:10:48.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.495+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40546e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.495+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4055160 con 0x7fcfe4054300 2024-04-27T04:10:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.495+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8001200 tx=0x7fcfe800b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:10:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.496+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4054300 2024-04-27T04:10:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.497+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4054300 2024-04-27T04:10:48.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:48.497+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe801e950 con 0x7fcfe4054300 2024-04-27T04:10:48.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:10:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:10:48.494+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 2098 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:10:48.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:48.353+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:48.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:48.353+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:48.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:48.353+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:48.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:48.354+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:49.353 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:48.972+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:49.353 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:48.972+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:49.353 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:48.972+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:49.353 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:48.973+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:49.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:49.064+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:49.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:49.064+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:49.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:49.064+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:49.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:49.065+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:49.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:49.353+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:49.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:49.353+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:49.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:49.353+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:49.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:49.354+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:50.370 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:49.940+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:50.371 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:49.940+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:50.371 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:49.940+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:50.371 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:49.941+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:50.371 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:50.023+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:50.371 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:50.023+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:50.372 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:50.023+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:50.372 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:50.023+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:50.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:50.371+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:50.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:50.371+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:50.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:50.371+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:50.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:50.372+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:50.992 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:50.934+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:50.993 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:50.934+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:50.993 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:50.934+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:50.993 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:50.935+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:51.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:50.993+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:51.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:50.993+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:51.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:50.993+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:51.354 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:50.995+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:51.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:51.354+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:51.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:51.354+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:51.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:51.354+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:51.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:51.355+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:52.389 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:51.957+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:52.389 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:51.957+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:52.389 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:51.957+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:52.389 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:51.958+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:52.389 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:51.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:52.390 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:51.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:52.390 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:51.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:52.390 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:51.962+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:52.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:52.390+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:52.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:52.390+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:52.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:52.390+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:52.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:52.391+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:53.205 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:52.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:53.205 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:52.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:53.205 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:52.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:53.205 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:52.979+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:53.205 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:52.935+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:53.205 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:52.935+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:53.205 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:52.935+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:53.205 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:52.936+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:53.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.494+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:10:53.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.495+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:10:53.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.495+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8001200 tx=0x7fcfe800b940 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:10:53.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.495+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8001200 tx=0x7fcfe800b940 comp rx=0 tx=0).stop 2024-04-27T04:10:53.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.495+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:10:53.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.495+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:53.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.495+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:53.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.495+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:53.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.495+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051c20 con 0x7fcfe4044cf0 2024-04-27T04:10:53.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.495+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f4b0 con 0x7fcff8101eb0 2024-04-27T04:10:53.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.495+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:10:53.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.496+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:10:53.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.496+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:53.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.496+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052990 con 0x7fcfe4044cf0 2024-04-27T04:10:53.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.496+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fcff0039070 tx=0x7fcfe803a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:10:53.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.497+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0019c80 con 0x7fcfe4044cf0 2024-04-27T04:10:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.498+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff004d120 con 0x7fcfe4044cf0 2024-04-27T04:10:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:53.499+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4044cf0 2024-04-27T04:10:53.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:10:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:10:53.494+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 2160 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:10:53.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:53.380+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:53.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:53.380+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:53.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:53.380+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:53.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:53.381+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:54.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:54.000+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:54.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:54.000+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:54.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:54.000+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:54.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:54.001+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:54.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:53.895+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:54.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:53.895+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:54.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:53.895+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:54.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:53.895+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:54.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:54.369+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:54.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:54.369+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:54.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:54.369+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:54.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:54.371+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:55.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:55.008+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:55.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:55.008+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:55.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:55.008+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:55.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:55.009+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:54.909+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:54.909+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:55.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:54.909+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:55.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:54.910+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:55.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:55.378+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:55.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:55.378+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:55.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:55.378+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:55.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:55.379+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:56.031 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:55.955+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:56.031 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:55.955+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:56.031 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:55.955+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:56.032 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:55.956+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:56.376 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:56.032+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:56.376 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:56.032+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:56.377 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:56.032+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:56.377 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:56.033+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:56.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:56.377+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:56.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:56.377+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:56.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:56.377+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:56.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:56.378+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:57.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:57.045+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:57.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:57.045+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:57.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:57.045+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:57.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:57.045+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:57.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:56.913+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:57.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:56.913+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:57.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:56.913+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:57.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:56.914+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:57.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:57.358+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:57.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:57.358+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:57.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:57.358+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:57.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:57.359+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:58.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:58.004+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:58.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:58.004+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:58.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:58.004+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:58.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:58.004+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:58.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:57.883+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:58.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:57.883+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:58.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:57.883+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:58.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:57.884+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:58.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.495+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:10:58.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.495+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:10:58.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.495+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fcff0039070 tx=0x7fcfe803a2c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:10:58.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.495+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fcff0039070 tx=0x7fcfe803a2c0 comp rx=0 tx=0).stop 2024-04-27T04:10:58.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.495+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:10:58.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.495+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:58.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.495+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:58.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.495+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:10:58.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.495+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f4b0 con 0x7fcfe4054300 2024-04-27T04:10:58.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.495+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404deb0 con 0x7fcff8101eb0 2024-04-27T04:10:58.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.497+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:10:58.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.497+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:10:58.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.497+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:10:58.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.497+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052990 con 0x7fcfe4054300 2024-04-27T04:10:58.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.497+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8015f10 tx=0x7fcff0028040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:10:58.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.499+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4054300 2024-04-27T04:10:58.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.499+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4054300 2024-04-27T04:10:58.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:10:58.499+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013510 con 0x7fcfe4054300 2024-04-27T04:10:58.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:10:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:10:58.495+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 2223 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:10:58.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:58.336+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:58.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:58.336+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:58.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:58.336+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:58.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:58.338+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:10:59.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:59.024+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:59.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:59.024+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:59.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:59.024+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:10:59.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:10:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:10:59.025+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:59.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:58.914+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:59.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:58.914+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:59.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:58.914+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:10:59.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:58.915+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:10:59.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:59.302+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:59.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:59.302+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:59.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:59.302+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:10:59.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:10:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:10:59.303+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:00.255 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:00.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:00.255 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:00.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:00.255 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:00.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:00.255 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:00.052+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:00.256 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:59.952+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:00.256 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:59.952+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:00.256 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:59.952+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:00.256 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:10:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:10:59.952+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:00.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:00.255+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:00.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:00.255+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:00.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:00.255+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:00.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:00.256+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:01.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:01.057+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:01.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:01.057+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:01.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:01.057+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:01.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:01.058+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:01.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:00.935+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:01.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:00.935+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:01.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:00.935+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:01.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:00.936+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:01.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:01.226+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:01.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:01.226+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:01.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:01.226+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:01.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:01.226+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:02.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:02.024+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:02.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:02.024+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:02.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:02.024+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:02.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:02.024+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:02.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:01.889+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:02.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:01.889+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:02.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:01.889+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:02.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:01.890+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:02.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:02.420+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:11:02.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:02.420+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:11:02.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:02.268+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:02.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:02.268+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:02.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:02.268+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:02.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:02.270+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:03.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:03.038+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:03.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:03.038+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:03.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:03.038+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:03.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:03.040+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:03.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:02.892+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:03.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:02.892+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:03.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:02.892+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:03.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:02.893+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:03.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.494+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:11:03.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.494+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:11:03.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.494+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8015f10 tx=0x7fcff0028040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:11:03.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.494+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8015f10 tx=0x7fcff0028040 comp rx=0 tx=0).stop 2024-04-27T04:11:03.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.494+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:11:03.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.494+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:03.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.494+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:03.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.494+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:03.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.494+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404f4b0 con 0x7fcfe4044cf0 2024-04-27T04:11:03.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.494+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404deb0 con 0x7fcff8101eb0 2024-04-27T04:11:03.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.494+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:11:03.496 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:03.243+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:03.496 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:03.243+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:03.496 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:03.243+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:03.496 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:03.244+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.495+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:11:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.495+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.495+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052990 con 0x7fcfe4044cf0 2024-04-27T04:11:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.496+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fcff0032010 tx=0x7fcfe804fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:11:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.496+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00046c0 con 0x7fcfe4044cf0 2024-04-27T04:11:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.497+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff003aa90 con 0x7fcfe4044cf0 2024-04-27T04:11:03.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:03.498+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff003ed60 con 0x7fcfe4044cf0 2024-04-27T04:11:03.886 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:11:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:11:03.495+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 2284 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:11:04.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:03.886+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:04.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:03.886+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:04.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:03.886+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:04.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:03.887+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:04.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:04.000+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:04.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:04.000+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:04.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:04.000+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:04.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:04.001+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:04.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:04.200+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:04.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:04.200+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:04.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:04.200+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:04.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:04.201+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:05.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:04.869+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:05.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:04.869+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:05.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:04.869+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:05.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:04.870+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:05.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:05.025+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:05.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:05.025+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:05.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:05.025+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:05.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:05.025+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:05.492 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:05.194+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:05.492 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:05.194+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:05.492 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:05.194+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:05.492 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:05.195+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:06.165 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:05.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:06.165 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:05.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:06.165 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:05.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:06.165 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:05.834+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:06.166 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:06.028+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:06.166 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:06.028+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:06.166 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:06.028+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:06.166 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:06.028+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:06.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:06.164+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:06.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:06.164+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:06.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:06.164+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:06.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:06.166+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:07.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:07.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:07.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:07.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:07.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:07.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:07.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:07.012+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:07.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:06.789+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:07.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:06.789+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:07.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:06.789+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:07.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:06.790+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:07.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:07.184+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:07.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:07.184+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:07.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:07.184+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:07.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:07.185+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:08.143 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:07.838+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:08.143 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:07.838+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:08.143 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:07.838+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:08.143 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:07.840+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:08.143 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:07.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:08.144 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:07.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:08.144 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:07.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:08.144 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:07.979+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:08.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:08.143+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:08.143+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:08.143+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:08.144+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:08.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.495+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:11:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.495+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:11:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.495+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fcff0032010 tx=0x7fcfe804fa10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:11:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.495+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fcff0032010 tx=0x7fcfe804fa10 comp rx=0 tx=0).stop 2024-04-27T04:11:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.495+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:11:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.495+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.495+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.495+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.495+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404deb0 con 0x7fcfe4054300 2024-04-27T04:11:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.495+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcff8101eb0 2024-04-27T04:11:08.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.496+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:11:08.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.496+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:11:08.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.496+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:08.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.496+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052ad0 con 0x7fcfe4054300 2024-04-27T04:11:08.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.497+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400e330 tx=0x7fcff0009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:11:08.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.497+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00046c0 con 0x7fcfe4054300 2024-04-27T04:11:08.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.497+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0013ce0 con 0x7fcfe4054300 2024-04-27T04:11:08.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:08.498+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4054300 2024-04-27T04:11:08.836 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:11:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:11:08.495+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 2346 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:11:09.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:08.836+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:09.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:08.836+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:09.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:08.836+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:09.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:08.837+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:09.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:09.015+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:09.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:09.015+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:09.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:09.015+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:09.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:09.015+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:09.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:09.175+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:09.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:09.175+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:09.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:09.175+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:09.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:09.176+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:10.130 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:09.863+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:10.130 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:09.863+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:10.130 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:09.863+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:10.130 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:09.864+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:10.130 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:10.055+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:10.130 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:10.055+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:10.131 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:10.055+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:10.131 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:10.056+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:10.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:10.130+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:10.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:10.130+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:10.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:10.130+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:10.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:10.131+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:11.146 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:11.030+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:11.146 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:11.030+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:11.146 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:11.030+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:11.146 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:11.030+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:11.147 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:10.853+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:11.147 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:10.853+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:11.147 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:10.853+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:11.147 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:10.854+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:11.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:11.147+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:11.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:11.147+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:11.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:11.147+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:11.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:11.147+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:12.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:11.984+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:12.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:11.984+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:12.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:11.984+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:12.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:11.985+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:11.863+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:11.863+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:11.863+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:11.864+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:12.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:12.193+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:12.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:12.193+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:12.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:12.193+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:12.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:12.193+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:13.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:13.034+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:13.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:13.034+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:13.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:13.034+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:13.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:13.034+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:13.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:12.866+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:13.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:12.866+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:13.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:12.866+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:13.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:12.866+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.496+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:11:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.496+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:11:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.496+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400e330 tx=0x7fcff0009fd0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:11:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.496+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400e330 tx=0x7fcff0009fd0 comp rx=0 tx=0).stop 2024-04-27T04:11:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.496+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:11:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.496+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.497+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.497+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.497+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcfe4044cf0 2024-04-27T04:11:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.497+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ad0 con 0x7fcff8101eb0 2024-04-27T04:11:13.497 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:13.229+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:13.497 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:13.229+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:13.497 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:13.229+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:13.497 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:13.229+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:13.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.497+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:11:13.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.498+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:11:13.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.498+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:13.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.498+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052c10 con 0x7fcfe4044cf0 2024-04-27T04:11:13.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.498+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fcff000dfe0 tx=0x7fcff0041f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:11:13.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.499+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00046c0 con 0x7fcfe4044cf0 2024-04-27T04:11:13.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.499+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0009330 con 0x7fcfe4044cf0 2024-04-27T04:11:13.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:13.500+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059040 con 0x7fcfe4044cf0 2024-04-27T04:11:13.893 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:11:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:11:13.497+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 2408 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:11:14.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:14.071+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:14.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:14.071+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:14.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:14.071+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:14.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:14.071+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:13.894+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:13.894+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:13.894+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:13.895+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:14.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:14.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:14.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:14.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:14.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:14.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:14.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:14.183+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:15.089 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:14.945+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:15.090 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:14.945+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:15.090 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:14.945+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:15.090 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:14.946+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:15.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:15.090+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:15.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:15.090+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:15.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:15.090+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:15.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:15.091+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:15.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:15.153+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:15.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:15.153+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:15.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:15.153+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:15.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:15.154+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:16.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:15.974+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:16.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:15.974+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:16.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:15.974+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:16.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:15.975+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:16.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:16.088+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:16.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:16.088+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:16.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:16.088+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:16.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:16.089+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:16.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:16.155+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:16.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:16.155+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:16.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:16.155+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:16.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:16.155+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:17.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:17.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:17.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:17.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:17.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:17.052+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:17.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:17.119+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:17.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:17.119+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:17.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:17.119+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:17.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:17.120+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:17.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:16.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:17.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:16.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:17.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:16.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:17.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:16.962+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:17.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:17.781+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:11:17.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:17.781+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:11:18.058 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:17.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:18.058 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:17.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:18.058 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:17.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:18.058 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:17.963+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:18.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:18.056+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:18.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:18.056+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:18.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:18.056+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:18.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:18.057+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:18.151+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:18.151+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:18.151+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:18.152+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:18.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.497+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:11:18.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.497+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:11:18.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.497+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fcff000dfe0 tx=0x7fcff0041f80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:11:18.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.497+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fcff000dfe0 tx=0x7fcff0041f80 comp rx=0 tx=0).stop 2024-04-27T04:11:18.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.498+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:11:18.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.498+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:18.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.498+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:18.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.498+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:18.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.498+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcfe4054300 2024-04-27T04:11:18.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.498+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ad0 con 0x7fcff8101eb0 2024-04-27T04:11:18.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.498+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:11:18.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.499+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:11:18.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.499+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:18.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.499+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052c10 con 0x7fcfe4054300 2024-04-27T04:11:18.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.499+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8015f40 tx=0x7fcff0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:11:18.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.500+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4054300 2024-04-27T04:11:18.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.500+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4054300 2024-04-27T04:11:18.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:18.501+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013510 con 0x7fcfe4054300 2024-04-27T04:11:18.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:11:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:11:18.498+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 2471 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:11:19.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:18.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:19.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:18.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:19.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:18.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:19.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:18.998+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:19.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:19.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:19.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:19.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:19.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:19.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:19.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:19.063+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:19.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:19.163+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:19.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:19.163+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:19.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:19.163+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:19.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:19.163+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:20.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:20.027+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:20.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:20.027+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:20.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:20.027+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:20.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:20.028+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:20.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:20.013+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:20.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:20.013+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:20.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:20.013+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:20.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:20.014+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:20.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:20.126+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:20.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:20.126+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:20.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:20.126+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:20.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:20.127+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:21.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:20.998+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:21.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:20.998+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:21.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:20.998+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:21.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:20.999+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:21.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:21.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:21.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:21.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:21.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:21.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:21.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:21.037+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:21.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:21.160+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:21.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:21.160+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:21.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:21.160+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:21.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:21.161+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:22.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:22.028+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:22.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:22.028+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:22.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:22.028+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:22.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:22.029+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:22.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:22.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:22.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:22.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:22.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:22.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:22.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:22.023+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:22.126+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:22.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:22.126+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:22.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:22.126+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:22.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:22.127+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:23.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:22.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:23.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:22.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:23.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:22.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:23.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:22.993+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:23.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:22.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:23.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:22.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:23.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:22.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:23.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:22.979+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:23.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:23.126+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:23.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:23.126+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:23.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:23.126+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:23.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:23.127+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:23.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.498+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:11:23.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.498+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:11:23.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.498+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8015f40 tx=0x7fcff0000bc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:11:23.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.498+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8015f40 tx=0x7fcff0000bc0 comp rx=0 tx=0).stop 2024-04-27T04:11:23.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.498+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:11:23.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.498+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:23.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.498+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:23.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.498+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:23.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.498+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcfe4044cf0 2024-04-27T04:11:23.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.498+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ad0 con 0x7fcff8101eb0 2024-04-27T04:11:23.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.499+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:11:23.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.499+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:11:23.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.499+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:23.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.499+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052c10 con 0x7fcfe4044cf0 2024-04-27T04:11:23.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.500+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8002280 tx=0x7fcfe8054860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:11:23.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.500+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4044cf0 2024-04-27T04:11:23.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.501+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe801bab0 con 0x7fcfe4044cf0 2024-04-27T04:11:23.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:23.502+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013510 con 0x7fcfe4044cf0 2024-04-27T04:11:23.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:11:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:11:23.498+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 2534 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:11:24.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:23.961+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:24.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:23.961+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:24.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:23.961+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:24.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:23.961+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:24.121+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:24.121+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:24.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:24.121+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:24.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:24.122+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:24.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:23.958+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:24.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:23.958+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:24.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:23.958+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:24.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:23.959+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:25.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:24.967+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:25.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:24.967+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:25.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:24.967+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:25.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:24.968+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:25.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:24.993+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:25.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:24.993+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:25.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:24.993+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:25.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:24.994+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:25.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:25.135+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:25.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:25.135+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:25.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:25.135+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:25.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:25.136+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:26.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:25.948+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:26.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:25.948+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:26.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:25.948+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:26.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:25.948+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:26.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:26.174+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:26.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:26.174+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:26.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:26.174+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:26.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:26.175+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:26.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:26.030+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:26.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:26.030+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:26.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:26.030+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:26.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:26.031+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:27.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:26.921+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:27.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:26.921+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:27.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:26.921+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:27.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:26.921+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:27.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:27.142+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:27.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:27.142+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:27.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:27.142+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:27.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:27.143+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:27.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:27.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:27.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:27.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:27.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:27.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:27.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:27.023+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:28.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:28.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:28.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:28.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:28.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:28.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:28.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:28.071+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:28.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:27.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:28.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:27.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:28.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:27.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:28.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:27.914+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.499+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:11:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.499+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:11:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.499+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8002280 tx=0x7fcfe8054860 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:11:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.499+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8002280 tx=0x7fcfe8054860 comp rx=0 tx=0).stop 2024-04-27T04:11:28.499 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:28.187+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:28.499 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:28.187+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:28.499 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:28.187+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:28.499 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:28.188+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.499+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:11:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.499+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.499+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.499+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.499+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4051c20 con 0x7fcfe4054300 2024-04-27T04:11:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.499+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcff8101eb0 2024-04-27T04:11:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.499+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:11:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.500+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:11:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.500+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.500+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052ad0 con 0x7fcfe4054300 2024-04-27T04:11:28.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.500+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009de0 tx=0x7fcfe8042d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:11:28.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.501+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4054300 2024-04-27T04:11:28.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.502+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4054300 2024-04-27T04:11:28.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:28.503+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013510 con 0x7fcfe4054300 2024-04-27T04:11:28.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:11:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:11:28.499+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 2599 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:11:29.230 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:29.102+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:29.231 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:29.102+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:29.231 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:29.102+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:29.231 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:29.102+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:29.231 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:28.923+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:29.231 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:28.923+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:29.231 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:28.923+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:29.231 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:28.923+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:29.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:29.231+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:29.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:29.231+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:29.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:29.231+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:29.233+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:30.251 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:29.937+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:30.251 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:29.938+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:30.251 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:29.938+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:30.251 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:29.938+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:30.251 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:30.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:30.251 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:30.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:30.251 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:30.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:30.252 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:30.106+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:30.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:30.251+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:30.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:30.251+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:30.251+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:30.253+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:31.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:30.918+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:31.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:30.918+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:31.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:30.918+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:31.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:30.919+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:31.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:31.127+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:31.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:31.127+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:31.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:31.127+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:31.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:31.127+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:31.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:31.268+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:31.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:31.268+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:31.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:31.268+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:31.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:31.269+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:32.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:32.122+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:32.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:32.122+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:32.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:32.122+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:32.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:32.122+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:32.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:31.891+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:32.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:31.891+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:32.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:31.891+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:32.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:31.892+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:32.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:32.240+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:32.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:32.240+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:32.240+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:32.241+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:33.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:33.119+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:33.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:33.119+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:33.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:33.119+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:33.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:33.120+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:33.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:32.899+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:33.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:32.899+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:33.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:32.899+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:33.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:32.900+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:33.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.499+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:11:33.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.499+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:11:33.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.499+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009de0 tx=0x7fcfe8042d30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:11:33.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.499+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009de0 tx=0x7fcfe8042d30 comp rx=0 tx=0).stop 2024-04-27T04:11:33.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.500+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:11:33.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.500+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:33.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.500+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:33.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.500+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:33.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.500+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcfe4044cf0 2024-04-27T04:11:33.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.500+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ad0 con 0x7fcff8101eb0 2024-04-27T04:11:33.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.500+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:11:33.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.500+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:11:33.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.501+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:33.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.501+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052c10 con 0x7fcfe4044cf0 2024-04-27T04:11:33.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.501+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fcff000e540 tx=0x7fcfe8031a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:11:33.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.501+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003a340 con 0x7fcfe4044cf0 2024-04-27T04:11:33.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.501+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff000ea30 con 0x7fcfe4044cf0 2024-04-27T04:11:33.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:33.503+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059030 con 0x7fcfe4044cf0 2024-04-27T04:11:33.536 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:11:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:11:33.500+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 2664 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:11:33.537 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:33.257+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:33.537 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:33.257+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:33.537 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:33.257+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:33.537 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:33.259+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:34.217 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:33.927+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:34.217 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:33.927+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:34.218 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:33.927+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:34.218 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:33.928+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:34.218 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:34.124+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:34.218 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:34.124+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:34.218 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:34.124+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:34.218 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:34.125+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:34.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:34.218+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:34.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:34.218+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:34.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:34.218+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:34.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:34.219+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:35.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:34.921+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:35.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:34.921+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:35.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:34.921+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:35.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:34.922+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:35.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:35.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:35.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:35.134+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:35.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:35.217+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:35.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:35.217+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:35.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:35.217+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:35.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:35.217+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:36.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:35.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:36.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:35.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:36.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:35.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:36.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:35.913+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:36.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:36.116+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:36.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:36.116+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:36.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:36.116+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:36.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:36.117+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:36.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:36.278+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:11:36.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:36.278+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:11:36.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:36.243+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:36.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:36.243+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:36.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:36.243+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:36.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:36.244+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:37.145 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:36.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:37.145 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:36.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:37.145 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:36.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:37.145 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:36.913+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:37.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:37.146+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:37.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:37.146+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:37.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:37.146+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:37.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:37.147+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:37.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:37.243+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:37.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:37.243+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:37.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:37.243+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:37.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:37.243+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:38.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:37.917+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:38.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:37.917+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:38.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:37.917+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:38.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:37.918+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:38.500 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:38.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:38.500 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:38.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:38.500 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:38.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:38.500 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:38.183+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:38.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.500+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-27T04:11:38.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.500+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:11:38.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.500+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fcff000e540 tx=0x7fcfe8031a10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:11:38.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.500+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fcff000e540 tx=0x7fcfe8031a10 comp rx=0 tx=0).stop 2024-04-27T04:11:38.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.500+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:11:38.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.500+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:38.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.500+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:38.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.500+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:38.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.500+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ad0 con 0x7fcfe4054300 2024-04-27T04:11:38.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.500+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c10 con 0x7fcff8101eb0 2024-04-27T04:11:38.501 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:38.273+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:38.501 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:38.273+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:38.501 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:38.273+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:38.501 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:38.274+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:38.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.501+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:11:38.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.501+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:11:38.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.501+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:38.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.501+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052d50 con 0x7fcfe4054300 2024-04-27T04:11:38.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.502+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fcff0035fa0 tx=0x7fcff0009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:11:38.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.503+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003a340 con 0x7fcfe4054300 2024-04-27T04:11:38.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.503+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0041c10 con 0x7fcfe4054300 2024-04-27T04:11:38.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:38.505+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059030 con 0x7fcfe4054300 2024-04-27T04:11:38.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:11:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:11:38.501+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 2737 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:11:39.224 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:38.954+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:39.224 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:38.954+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:39.224 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:38.954+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:39.224 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:38.955+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:39.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:39.317+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:39.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:39.317+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:39.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:39.317+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:39.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:39.318+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:39.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:39.225+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:39.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:39.225+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:39.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:39.225+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:39.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:39.225+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:40.229 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:39.941+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:40.229 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:39.941+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:40.230 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:39.941+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:40.230 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:39.941+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:40.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:40.285+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:40.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:40.285+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:40.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:40.285+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:40.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:40.286+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:40.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:40.230+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:40.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:40.230+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:40.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:40.230+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:40.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:40.231+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:41.265 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:40.982+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:41.265 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:40.982+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:41.266 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:40.982+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:41.266 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:40.983+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:41.266 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:41.213+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:41.266 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:41.213+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:41.266 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:41.213+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:41.266 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:41.214+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:41.568 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:41.296+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:41.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:41.296+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:41.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:41.296+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:41.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:41.297+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:42.331 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:42.018+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:42.331 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:42.018+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:42.331 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:42.018+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:42.331 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:42.019+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:42.331 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:42.168+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:42.331 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:42.168+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:42.331 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:42.168+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:42.331 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:42.169+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:42.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:42.331+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:42.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:42.331+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:42.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:42.331+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:42.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:42.333+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:43.357 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:43.180+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:43.357 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:43.180+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:43.357 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:43.180+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:43.357 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:43.181+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:43.357 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:43.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:43.358 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:43.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:43.358 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:43.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:43.358 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:43.041+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:43.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.501+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:11:43.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.501+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:11:43.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.501+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fcff0035fa0 tx=0x7fcff0009fd0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:11:43.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.501+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fcff0035fa0 tx=0x7fcff0009fd0 comp rx=0 tx=0).stop 2024-04-27T04:11:43.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.502+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:11:43.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.502+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:43.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.502+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:43.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.502+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:43.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.502+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052a40 con 0x7fcfe4044cf0 2024-04-27T04:11:43.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.502+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b80 con 0x7fcff8101eb0 2024-04-27T04:11:43.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.502+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:11:43.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.503+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:11:43.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.503+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:43.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.503+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052cc0 con 0x7fcfe4044cf0 2024-04-27T04:11:43.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.503+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fcff0018ff0 tx=0x7fcff0032900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:11:43.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.505+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4044cf0 2024-04-27T04:11:43.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.505+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff000e400 con 0x7fcfe4044cf0 2024-04-27T04:11:43.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:43.505+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4044cf0 2024-04-27T04:11:43.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:11:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:11:43.502+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 2798 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:11:43.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:43.358+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:43.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:43.358+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:43.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:43.358+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:43.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:43.359+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:44.353 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:44.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:44.353 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:44.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:44.353 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:44.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:44.353 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:44.204+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:44.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:44.049+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:44.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:44.049+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:44.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:44.049+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:44.354 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:44.051+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:44.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:44.353+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:44.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:44.353+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:44.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:44.353+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:44.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:44.355+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:45.309 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:45.237+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:45.310 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:45.237+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:45.310 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:45.237+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:45.310 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:45.237+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:45.310 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:45.019+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:45.311 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:45.019+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:45.311 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:45.019+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:45.311 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:45.020+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:45.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:45.310+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:45.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:45.310+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:45.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:45.310+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:45.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:45.311+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:46.294 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:46.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:46.295 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:46.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:46.295 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:46.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:46.295 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:46.199+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:46.295 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:45.971+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:46.295 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:45.971+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:46.295 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:45.971+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:46.295 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:45.972+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:46.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:46.295+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:46.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:46.295+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:46.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:46.295+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:46.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:46.296+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:47.279 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:46.982+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:47.279 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:46.982+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:47.279 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:46.982+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:47.279 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:46.983+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:47.279 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:47.228+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:47.279 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:47.228+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:47.279 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:47.228+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:47.279 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:47.229+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:47.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:47.280+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:47.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:47.280+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:47.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:47.280+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:47.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:47.281+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:48.275 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:47.939+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:48.275 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:47.939+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:48.275 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:47.939+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:48.275 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:47.940+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:48.275 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:48.179+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:48.276 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:48.179+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:48.276 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:48.179+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:48.276 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:48.180+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:48.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.502+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:11:48.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.502+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:11:48.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.502+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fcff0018ff0 tx=0x7fcff0032900 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:11:48.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.502+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fcff0018ff0 tx=0x7fcff0032900 comp rx=0 tx=0).stop 2024-04-27T04:11:48.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.503+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:11:48.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.503+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:48.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.503+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:48.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.503+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:48.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.503+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcfe4054300 2024-04-27T04:11:48.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.503+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ad0 con 0x7fcff8101eb0 2024-04-27T04:11:48.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.503+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:11:48.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.504+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:11:48.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.504+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:48.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.504+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052c10 con 0x7fcfe4054300 2024-04-27T04:11:48.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.504+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fcff000ee90 tx=0x7fcff0019440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:11:48.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.505+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4054300 2024-04-27T04:11:48.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.506+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0025850 con 0x7fcfe4054300 2024-04-27T04:11:48.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:48.506+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059040 con 0x7fcfe4054300 2024-04-27T04:11:48.565 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:11:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:11:48.503+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 2860 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:11:48.565 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:48.275+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:48.565 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:48.275+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:48.565 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:48.275+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:48.565 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:48.277+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:49.297 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:49.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:49.297 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:49.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:49.298 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:49.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:49.298 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:49.176+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:49.298 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:48.950+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:49.298 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:48.950+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:49.298 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:48.950+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:49.298 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:48.951+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:49.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:49.298+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:49.737 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:49.298+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:49.737 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:49.298+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:49.737 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:49.299+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:50.257 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:50.000+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:50.258 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:50.000+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:50.258 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:50.000+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:50.258 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:50.001+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:50.258 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:50.126+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:50.258 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:50.127+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:50.258 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:50.127+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:50.258 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:50.128+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:50.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:50.258+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:50.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:50.258+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:50.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:50.258+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:50.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:50.259+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:51.374 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:51.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:51.374 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:51.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:51.374 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:51.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:51.374 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:51.024+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:51.374 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:51.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:51.374 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:51.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:51.374 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:51.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:51.374 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:51.087+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:51.375 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:51.266+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:51.375 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:51.266+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:51.375 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:51.266+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:51.375 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:51.267+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:51.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:51.637+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:11:51.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:51.637+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:11:52.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:52.134+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:52.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:52.134+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:52.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:52.134+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:52.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:52.135+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:52.060+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:52.060+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:52.060+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:52.060+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:52.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:52.234+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:52.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:52.234+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:52.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:52.234+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:52.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:52.235+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:53.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:53.097+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:53.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:53.097+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:53.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:53.097+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:53.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:53.098+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:53.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:53.085+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:53.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:53.085+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:53.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:53.085+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:53.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:53.086+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:53.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:53.232+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:53.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:53.232+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:53.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:53.232+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:53.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:53.232+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:53.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.503+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:11:53.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.503+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:11:53.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.503+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fcff000ee90 tx=0x7fcff0019440 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:11:53.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.503+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fcff000ee90 tx=0x7fcff0019440 comp rx=0 tx=0).stop 2024-04-27T04:11:53.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.503+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:11:53.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.503+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:53.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.503+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:53.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.503+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:53.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.503+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcfe4044cf0 2024-04-27T04:11:53.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.503+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ad0 con 0x7fcff8101eb0 2024-04-27T04:11:53.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.504+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:11:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.504+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:11:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.504+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.504+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052c10 con 0x7fcfe4044cf0 2024-04-27T04:11:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.505+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800b020 tx=0x7fcff0028040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:11:53.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.506+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4044cf0 2024-04-27T04:11:53.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.506+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4044cf0 2024-04-27T04:11:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:53.507+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013510 con 0x7fcfe4044cf0 2024-04-27T04:11:53.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:11:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:11:53.503+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 2923 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:11:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:54.126+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:54.126+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:54.126+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:54.127+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:54.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:54.217+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:54.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:54.217+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:54.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:54.217+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:54.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:54.218+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:54.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:54.054+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:54.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:54.054+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:54.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:54.054+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:54.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:54.054+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:55.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:55.099+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:55.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:55.099+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:55.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:55.099+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:55.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:55.099+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:55.203+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:55.203+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:55.203+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:55.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:55.204+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:55.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:55.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:55.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:55.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:55.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:55.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:55.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:55.017+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:56.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:56.136+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:56.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:56.136+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:56.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:56.136+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:56.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:56.138+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:56.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:56.244+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:56.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:56.244+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:56.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:56.244+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:56.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:56.245+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:56.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:56.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:56.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:56.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:56.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:56.003+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:56.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:56.003+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:57.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:57.172+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:57.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:57.172+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:57.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:57.172+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:57.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:57.174+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:57.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:57.221+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:57.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:57.221+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:57.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:57.221+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:57.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:57.222+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:57.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:57.045+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:57.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:57.045+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:57.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:57.045+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:57.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:57.046+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:58.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:58.228+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:58.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:58.228+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:58.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:58.228+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:58.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:58.228+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:58.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:58.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:58.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:58.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:58.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:58.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:58.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:58.024+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:58.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:58.172+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:58.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:58.172+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:58.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:58.172+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:58.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:58.174+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.503+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:11:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.504+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:11:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.504+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800b020 tx=0x7fcff0028040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:11:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.504+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800b020 tx=0x7fcff0028040 comp rx=0 tx=0).stop 2024-04-27T04:11:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.504+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:11:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.504+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.504+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.504+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:11:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.504+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcfe4054300 2024-04-27T04:11:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.504+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ad0 con 0x7fcff8101eb0 2024-04-27T04:11:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.504+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:11:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.505+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:11:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.505+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:11:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.505+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052c10 con 0x7fcfe4054300 2024-04-27T04:11:58.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.505+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8033cf0 tx=0x7fcfe8043a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:11:58.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.506+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4054300 2024-04-27T04:11:58.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.507+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe801e780 con 0x7fcfe4054300 2024-04-27T04:11:58.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:11:58.507+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013510 con 0x7fcfe4054300 2024-04-27T04:11:58.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:11:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:11:58.504+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 2987 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:11:59.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:59.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:59.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:59.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:59.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:59.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:11:59.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:11:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:11:59.236+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:11:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:59.011+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:59.011+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:59.011+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:11:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:59.012+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:11:59.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:59.136+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:59.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:59.136+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:59.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:59.136+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:11:59.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:11:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:11:59.136+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:00.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:00.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:00.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:00.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:00.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:00.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:00.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:00.130+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:00.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:00.201+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:00.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:00.201+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:00.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:00.201+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:00.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:00.201+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:00.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:59.967+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:00.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:59.967+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:00.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:59.967+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:00.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:11:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:11:59.968+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:01.322 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:01.105+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:01.323 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:01.105+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:01.323 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:01.105+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:01.323 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:01.106+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:01.323 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:01.158+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:01.323 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:01.158+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:01.323 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:01.158+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:01.323 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:01.158+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:01.324 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:00.985+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:01.324 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:00.985+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:01.324 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:00.985+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:01.324 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:00.986+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:02.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:01.977+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:02.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:01.977+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:02.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:01.977+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:02.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:01.978+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:02.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:02.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:02.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:02.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:02.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:02.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:02.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:02.129+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:02.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:02.111+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:02.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:02.111+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:02.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:02.111+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:02.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:02.112+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:03.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:03.010+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:03.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:03.010+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:03.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:03.010+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:03.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:03.010+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:03.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:03.132+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:03.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:03.132+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:03.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:03.132+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:03.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:03.134+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:03.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:03.132+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:03.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:03.132+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:03.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:03.132+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:03.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:03.133+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:03.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.504+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:12:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.504+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:12:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.504+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8033cf0 tx=0x7fcfe8043a40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:12:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.504+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8033cf0 tx=0x7fcfe8043a40 comp rx=0 tx=0).stop 2024-04-27T04:12:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.504+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:12:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.504+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.504+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.504+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.504+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052af0 con 0x7fcfe4044cf0 2024-04-27T04:12:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.504+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c30 con 0x7fcff8101eb0 2024-04-27T04:12:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.505+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:12:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.506+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:12:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.506+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.506+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052d70 con 0x7fcfe4044cf0 2024-04-27T04:12:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.506+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fcff0022f50 tx=0x7fcfe800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:12:03.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.507+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4044cf0 2024-04-27T04:12:03.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.508+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0025850 con 0x7fcfe4044cf0 2024-04-27T04:12:03.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:03.509+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059040 con 0x7fcfe4044cf0 2024-04-27T04:12:03.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:12:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:12:03.504+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 3054 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:12:04.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:04.118+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:04.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:04.118+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:04.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:04.118+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:04.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:04.119+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:04.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:04.124+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:04.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:04.124+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:04.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:04.124+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:04.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:04.125+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:04.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:04.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:04.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:04.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:04.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:04.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:04.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:04.002+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:05.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:04.971+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:05.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:04.971+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:05.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:04.971+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:05.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:04.972+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:05.114+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:05.114+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:05.114+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:05.115+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:05.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:05.128+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:05.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:05.128+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:05.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:05.128+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:05.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:05.130+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:06.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:05.933+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:06.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:05.933+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:06.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:05.933+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:06.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:05.934+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:06.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:06.140+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:06.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:06.140+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:06.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:06.140+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:06.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:06.141+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:06.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:06.139+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:06.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:06.139+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:06.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:06.139+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:06.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:06.140+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:07.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:07.125+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:12:07.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:07.125+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:12:07.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:07.173+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:07.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:07.173+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:07.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:07.173+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:07.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:07.174+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:07.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:06.952+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:07.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:06.952+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:07.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:06.952+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:07.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:06.952+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:07.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:07.159+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:07.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:07.159+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:07.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:07.159+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:07.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:07.160+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:08.194 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:08.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:08.194 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:08.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:08.194 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:08.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:08.194 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:08.189+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:08.194 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:07.944+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:08.194 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:07.944+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:08.195 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:07.944+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:08.195 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:07.945+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:08.195 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:08.194+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:08.195 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:08.194+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:08.195 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:08.194+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:08.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.505+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:12:08.504 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:08.195+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:08.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.505+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:12:08.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.505+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fcff0022f50 tx=0x7fcfe800bda0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:12:08.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.505+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fcff0022f50 tx=0x7fcfe800bda0 comp rx=0 tx=0).stop 2024-04-27T04:12:08.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.505+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:12:08.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.505+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:08.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.505+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:08.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.505+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:08.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.505+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b40 con 0x7fcfe4054300 2024-04-27T04:12:08.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.505+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c80 con 0x7fcff8101eb0 2024-04-27T04:12:08.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.506+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:12:08.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.507+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:12:08.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.507+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:08.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.507+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052dc0 con 0x7fcfe4054300 2024-04-27T04:12:08.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.507+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8011f50 tx=0x7fcff005aa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:12:08.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.508+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4054300 2024-04-27T04:12:08.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.510+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8042f10 con 0x7fcfe4054300 2024-04-27T04:12:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:08.512+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013510 con 0x7fcfe4054300 2024-04-27T04:12:08.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:12:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:12:08.506+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 3132 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:12:09.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:08.948+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:09.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:08.948+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:09.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:08.948+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:09.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:08.948+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:09.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:09.176+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:09.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:09.176+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:09.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:09.176+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:09.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:09.178+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:09.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:09.240+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:09.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:09.240+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:09.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:09.240+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:09.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:09.241+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:10.287 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:09.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:10.287 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:09.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:10.287 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:09.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:10.287 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:09.987+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:10.287 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:10.204+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:10.288 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:10.204+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:10.288 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:10.204+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:10.288 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:10.204+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:10.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:10.288+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:10.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:10.288+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:10.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:10.288+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:10.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:10.289+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:11.332 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:11.013+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:11.332 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:11.013+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:11.333 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:11.013+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:11.333 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:11.014+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:11.333 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:11.228+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:11.333 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:11.228+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:11.333 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:11.228+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:11.333 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:11.229+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:11.608 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:11.333+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:11.608 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:11.333+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:11.608 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:11.333+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:11.608 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:11.334+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:12.374 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:12.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:12.374 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:12.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:12.374 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:12.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:12.374 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:12.215+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:12.374 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:12.057+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:12.374 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:12.057+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:12.374 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:12.057+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:12.375 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:12.058+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:12.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:12.375+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:12.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:12.375+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:12.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:12.375+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:12.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:12.375+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:13.402 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:13.211+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:13.403 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:13.211+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:13.403 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:13.211+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:13.403 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:13.211+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:13.403 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:13.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:13.403 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:13.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:13.403 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:13.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:13.403 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:13.076+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:13.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.506+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:12:13.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.506+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:12:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.506+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8011f50 tx=0x7fcff005aa30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:12:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.506+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8011f50 tx=0x7fcff005aa30 comp rx=0 tx=0).stop 2024-04-27T04:12:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.506+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:12:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.506+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.506+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.506+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.506+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b40 con 0x7fcfe4044cf0 2024-04-27T04:12:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.506+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c80 con 0x7fcff8101eb0 2024-04-27T04:12:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.507+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:12:13.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.507+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:12:13.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.507+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:13.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.507+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052dc0 con 0x7fcfe4044cf0 2024-04-27T04:12:13.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.508+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8031c40 tx=0x7fcfe802f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:12:13.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.508+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4044cf0 2024-04-27T04:12:13.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.508+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8042f10 con 0x7fcfe4044cf0 2024-04-27T04:12:13.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:13.509+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe802f1d0 con 0x7fcfe4044cf0 2024-04-27T04:12:13.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:12:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:12:13.506+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 3204 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:12:13.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:13.403+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:13.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:13.403+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:13.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:13.403+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:13.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:13.404+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:14.393 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:14.193+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:14.393 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:14.193+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:14.394 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:14.193+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:14.394 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:14.193+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:14.394 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:14.043+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:14.394 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:14.043+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:14.394 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:14.043+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:14.394 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:14.043+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:14.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:14.394+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:14.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:14.394+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:14.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:14.394+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:14.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:14.395+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:15.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:15.162+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:15.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:15.162+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:15.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:15.162+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:15.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:15.162+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:15.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:15.037+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:15.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:15.037+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:15.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:15.037+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:15.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:15.038+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:15.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:15.442+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:15.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:15.442+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:15.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:15.442+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:15.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:15.443+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:16.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:16.171+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:16.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:16.171+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:16.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:16.171+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:16.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:16.172+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:16.340 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:16.006+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:16.340 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:16.006+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:16.340 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:16.006+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:16.341 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:16.006+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:16.669 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:16.439+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:16.669 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:16.439+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:16.669 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:16.439+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:16.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:16.440+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:16.976+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:16.976+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:16.976+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:16.977+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:17.199+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:17.199+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:17.199+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:17.201+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:17.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:17.471+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:17.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:17.471+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:17.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:17.471+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:17.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:17.472+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:17.946+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:17.946+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:17.946+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:18.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:17.947+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:18.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:18.190+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:18.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:18.190+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:18.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:18.190+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:18.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:18.190+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:18.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.507+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:12:18.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.507+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:12:18.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.507+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8031c40 tx=0x7fcfe802f790 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:12:18.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.507+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8031c40 tx=0x7fcfe802f790 comp rx=0 tx=0).stop 2024-04-27T04:12:18.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.507+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:12:18.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.507+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:18.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.507+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:18.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.507+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.507+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b40 con 0x7fcfe4054300 2024-04-27T04:12:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.507+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c80 con 0x7fcff8101eb0 2024-04-27T04:12:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.507+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:12:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.508+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:12:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.508+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.508+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052dc0 con 0x7fcfe4054300 2024-04-27T04:12:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.508+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009680 tx=0x7fcfe8014c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:12:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.509+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4054300 2024-04-27T04:12:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.509+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe801e780 con 0x7fcfe4054300 2024-04-27T04:12:18.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:18.510+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013c60 con 0x7fcfe4054300 2024-04-27T04:12:18.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:12:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:12:18.507+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 3267 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:12:18.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:18.484+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:18.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:18.484+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:18.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:18.484+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:18.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:18.485+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:19.218 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:18.932+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:19.218 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:18.932+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:19.218 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:18.932+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:19.218 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:18.933+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:19.491 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:19.219+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:19.491 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:19.219+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:19.491 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:19.219+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:19.491 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:19.220+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:19.883 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:19.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:19.883 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:19.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:19.883 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:19.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:19.883 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:19.493+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:20.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:19.884+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:20.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:19.884+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:20.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:19.884+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:20.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:19.884+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:20.492 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:20.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:20.492 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:20.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:20.492 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:20.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:20.492 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:20.215+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:20.892 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:20.493+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:20.892 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:20.493+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:20.892 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:20.493+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:20.892 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:20.493+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:21.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:20.893+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:21.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:20.893+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:21.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:20.893+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:21.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:20.894+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:21.569 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:21.236+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:21.569 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:21.236+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:21.569 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:21.236+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:21.569 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:21.237+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:21.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:21.455+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:21.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:21.455+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:21.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:21.455+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:21.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:21.456+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:22.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:21.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:22.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:21.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:22.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:21.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:22.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:21.873+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:22.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:22.485+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:12:22.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:22.485+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:12:22.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:22.272+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:22.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:22.272+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:22.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:22.272+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:22.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:22.273+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:22.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:22.489+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:22.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:22.489+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:22.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:22.489+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:22.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:22.489+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:23.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:22.868+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:23.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:22.868+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:23.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:22.868+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:23.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:22.869+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.507+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:12:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.507+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:12:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.507+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009680 tx=0x7fcfe8014c80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:12:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.507+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009680 tx=0x7fcfe8014c80 comp rx=0 tx=0).stop 2024-04-27T04:12:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.507+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:12:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.508+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.508+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.508+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.508+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4046380 con 0x7fcfe4044cf0 2024-04-27T04:12:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.508+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcff8101eb0 2024-04-27T04:12:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.508+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:12:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.509+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:12:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.509+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.509+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052ad0 con 0x7fcfe4044cf0 2024-04-27T04:12:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.509+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fcff001ffc0 tx=0x7fcfe801bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:12:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.509+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4044cf0 2024-04-27T04:12:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.510+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0009420 con 0x7fcfe4044cf0 2024-04-27T04:12:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:23.511+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4044cf0 2024-04-27T04:12:23.539 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:12:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:12:23.508+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 3328 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:12:23.540 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:23.444+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:23.540 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:23.444+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:23.540 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:23.444+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:23.540 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:23.445+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:23.540 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:23.273+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:23.540 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:23.273+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:23.540 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:23.273+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:23.540 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:23.274+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:24.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:23.837+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:24.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:23.837+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:24.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:23.837+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:24.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:23.838+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:24.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:24.489+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:24.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:24.489+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:24.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:24.489+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:24.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:24.490+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:24.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:24.302+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:24.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:24.302+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:24.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:24.302+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:24.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:24.303+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:25.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:24.813+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:25.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:24.813+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:25.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:24.813+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:25.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:24.814+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:25.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:25.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:25.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:25.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:25.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:25.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:25.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:25.512+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:25.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:25.327+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:25.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:25.327+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:25.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:25.327+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:25.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:25.328+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:26.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:25.810+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:26.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:25.810+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:26.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:25.810+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:26.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:25.810+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:26.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:26.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:26.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:26.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:26.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:26.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:26.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:26.347+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:26.462+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:26.462+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:26.462+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:26.464+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:27.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:26.831+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:27.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:26.831+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:27.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:26.831+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:27.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:26.833+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:27.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:27.503+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:27.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:27.503+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:27.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:27.503+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:27.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:27.504+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:27.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:27.329+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:27.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:27.329+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:27.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:27.329+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:27.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:27.330+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:28.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:27.793+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:28.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:27.793+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:28.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:27.793+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:28.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:27.794+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:28.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.508+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:12:28.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.508+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:12:28.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.508+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fcff001ffc0 tx=0x7fcfe801bde0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:12:28.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.508+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fcff001ffc0 tx=0x7fcfe801bde0 comp rx=0 tx=0).stop 2024-04-27T04:12:28.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.508+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:12:28.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.508+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:28.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.508+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:28.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.508+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:28.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.508+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcfe4054300 2024-04-27T04:12:28.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.508+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ad0 con 0x7fcff8101eb0 2024-04-27T04:12:28.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.509+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:12:28.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.509+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:12:28.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.509+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:28.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.509+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052c10 con 0x7fcfe4054300 2024-04-27T04:12:28.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.509+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40184c0 tx=0x7fcff00369a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:12:28.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.510+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4054300 2024-04-27T04:12:28.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.510+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0009420 con 0x7fcfe4054300 2024-04-27T04:12:28.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:28.511+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4054300 2024-04-27T04:12:28.674 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:12:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:12:28.509+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 3390 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:12:28.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:28.459+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:28.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:28.459+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:28.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:28.459+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:28.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:28.460+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:28.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:28.364+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:28.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:28.364+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:28.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:28.364+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:28.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:28.365+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:29.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:28.810+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:29.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:28.810+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:29.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:28.810+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:29.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:28.811+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:29.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:29.496+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:29.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:29.496+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:29.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:29.496+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:29.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:29.497+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:29.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:29.336+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:29.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:29.336+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:29.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:29.336+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:29.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:29.336+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:30.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:29.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:30.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:29.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:30.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:29.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:30.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:29.833+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:30.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:30.506+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:30.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:30.506+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:30.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:30.506+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:30.506+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:30.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:30.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:30.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:30.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:30.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:30.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:30.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:30.347+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:31.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:30.857+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:31.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:30.857+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:31.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:30.857+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:31.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:30.858+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:31.474 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:31.313+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:31.475 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:31.313+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:31.475 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:31.313+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:31.475 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:31.314+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:31.825 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:31.476+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:31.826 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:31.476+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:31.826 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:31.476+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:31.826 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:31.477+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:32.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:31.844+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:32.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:31.844+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:32.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:31.844+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:32.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:31.844+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:32.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:32.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:32.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:32.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:32.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:32.517+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:32.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:32.313+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:32.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:32.313+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:32.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:32.313+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:32.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:32.313+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:33.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:32.860+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:33.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:32.860+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:33.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:32.860+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:33.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:32.861+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:33.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.508+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:12:33.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.508+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:12:33.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.508+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40184c0 tx=0x7fcff00369a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:12:33.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.508+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40184c0 tx=0x7fcff00369a0 comp rx=0 tx=0).stop 2024-04-27T04:12:33.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.508+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:12:33.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.508+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:33.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.508+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.508+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.508+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcfe4044cf0 2024-04-27T04:12:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.508+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ad0 con 0x7fcff8101eb0 2024-04-27T04:12:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.508+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:12:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.509+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:12:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.509+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.509+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052c10 con 0x7fcfe4044cf0 2024-04-27T04:12:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.509+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400dcc0 tx=0x7fcff003f620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:12:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.510+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4044cf0 2024-04-27T04:12:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.510+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe801e780 con 0x7fcfe4044cf0 2024-04-27T04:12:33.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:33.511+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe802f1d0 con 0x7fcfe4044cf0 2024-04-27T04:12:33.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:12:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:12:33.508+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 3452 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:12:33.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:33.566+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:33.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:33.566+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:33.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:33.566+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:33.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:33.567+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:33.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:33.332+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:33.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:33.332+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:33.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:33.332+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:33.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:33.333+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:34.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:33.816+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:34.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:33.816+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:34.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:33.816+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:34.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:33.817+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:34.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:34.524+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:34.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:34.524+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:34.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:34.524+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:34.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:34.525+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:34.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:34.340+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:34.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:34.340+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:34.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:34.340+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:34.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:34.341+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:35.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:34.793+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:35.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:34.794+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:35.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:34.794+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:34.794+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:35.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:35.304+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:35.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:35.304+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:35.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:35.304+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:35.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:35.305+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:35.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:35.794+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:35.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:35.794+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:35.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:35.794+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:35.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:35.795+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:35.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:35.568+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:35.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:35.568+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:35.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:35.568+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:35.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:35.568+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:36.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:36.309+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:36.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:36.309+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:36.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:36.309+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:36.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:36.310+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:36.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:36.529+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:36.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:36.529+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:36.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:36.529+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:36.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:36.530+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:37.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:36.837+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:37.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:36.837+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:37.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:36.837+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:37.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:36.837+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:37.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:37.484+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:37.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:37.484+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:37.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:37.484+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:37.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:37.484+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:37.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:37.288+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:37.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:37.288+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:37.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:37.288+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:37.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:37.288+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:37.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:37.844+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:12:37.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:37.844+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:12:38.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:37.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:38.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:37.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:38.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:37.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:38.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:37.872+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:38.469 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:38.469+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:38.469 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:38.264+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:38.469 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:38.264+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:38.470 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:38.264+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:38.470 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:38.265+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.508+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:12:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.509+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:12:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.509+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400dcc0 tx=0x7fcff003f620 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:12:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.509+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400dcc0 tx=0x7fcff003f620 comp rx=0 tx=0).stop 2024-04-27T04:12:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.509+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:12:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.509+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.509+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.509+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.509+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcfe4054300 2024-04-27T04:12:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.509+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ad0 con 0x7fcff8101eb0 2024-04-27T04:12:38.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.509+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:12:38.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.510+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:12:38.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.510+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:38.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.510+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052c10 con 0x7fcfe4054300 2024-04-27T04:12:38.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.510+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fcff0019de0 tx=0x7fcfe804fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:12:38.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.512+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4054300 2024-04-27T04:12:38.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.512+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff002bb30 con 0x7fcfe4054300 2024-04-27T04:12:38.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:38.513+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4054300 2024-04-27T04:12:38.872 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:12:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:12:38.509+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 3515 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:12:38.873 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:38.469+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:38.873 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:38.469+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:38.873 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:38.469+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:39.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:38.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:39.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:38.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:39.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:38.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:39.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:38.873+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:39.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:39.315+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:39.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:39.315+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:39.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:39.315+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:39.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:39.315+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:39.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:39.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:39.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:39.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:39.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:39.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:39.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:39.465+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:40.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:39.883+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:40.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:39.883+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:40.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:39.883+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:40.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:39.883+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:40.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:40.481+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:40.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:40.481+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:40.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:40.481+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:40.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:40.483+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:40.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:40.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:40.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:40.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:40.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:40.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:40.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:40.306+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:41.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:40.918+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:41.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:40.918+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:41.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:40.918+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:41.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:40.919+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:41.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:41.347+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:41.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:41.347+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:41.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:41.347+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:41.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:41.348+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:41.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:41.517+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:41.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:41.517+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:41.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:41.517+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:41.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:41.518+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:42.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:41.920+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:42.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:41.920+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:42.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:41.920+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:42.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:41.921+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:42.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:42.519+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:42.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:42.519+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:42.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:42.519+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:42.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:42.521+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:42.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:42.318+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:42.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:42.318+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:42.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:42.318+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:42.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:42.319+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:43.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:42.918+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:43.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:42.918+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:43.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:42.918+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:43.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:42.919+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.509+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:12:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.510+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:12:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.510+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fcff0019de0 tx=0x7fcfe804fa10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:12:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.510+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fcff0019de0 tx=0x7fcfe804fa10 comp rx=0 tx=0).stop 2024-04-27T04:12:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.510+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:12:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.510+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.510+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.510+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.510+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052a20 con 0x7fcfe4044cf0 2024-04-27T04:12:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.510+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b60 con 0x7fcff8101eb0 2024-04-27T04:12:43.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.510+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:12:43.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.511+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:12:43.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.511+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:43.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.511+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052ca0 con 0x7fcfe4044cf0 2024-04-27T04:12:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.512+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400c230 tx=0x7fcff0037950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:12:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.512+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003a340 con 0x7fcfe4044cf0 2024-04-27T04:12:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.512+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0023d40 con 0x7fcfe4044cf0 2024-04-27T04:12:43.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:43.513+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4044cf0 2024-04-27T04:12:43.540 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:12:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:12:43.510+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 3576 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:12:43.540 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:43.284+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:43.540 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:43.284+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:43.541 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:43.284+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:43.541 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:43.285+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:43.909 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:43.555+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:43.909 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:43.555+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:43.909 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:43.555+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:43.909 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:43.556+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:44.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:43.910+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:44.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:43.910+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:44.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:43.910+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:44.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:43.910+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:44.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:44.322+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:44.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:44.322+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:44.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:44.322+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:44.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:44.323+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:44.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:44.556+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:44.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:44.556+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:44.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:44.556+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:44.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:44.556+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:45.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:44.913+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:45.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:44.913+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:45.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:44.913+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:45.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:44.913+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:45.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:45.341+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:45.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:45.341+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:45.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:45.341+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:45.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:45.342+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:45.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:45.562+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:45.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:45.562+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:45.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:45.562+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:45.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:45.562+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:46.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:45.899+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:46.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:45.899+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:46.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:45.899+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:46.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:45.900+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:46.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:46.371+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:46.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:46.371+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:46.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:46.371+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:46.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:46.372+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:46.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:46.544+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:46.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:46.544+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:46.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:46.544+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:46.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:46.545+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:47.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:46.851+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:47.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:46.851+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:47.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:46.851+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:47.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:46.851+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:47.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:47.324+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:47.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:47.324+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:47.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:47.324+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:47.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:47.325+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:47.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:47.570+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:47.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:47.570+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:47.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:47.570+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:47.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:47.571+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:48.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:47.862+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:48.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:47.862+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:48.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:47.862+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:48.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:47.862+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.511+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:12:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.511+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:12:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.511+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400c230 tx=0x7fcff0037950 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:12:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.511+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400c230 tx=0x7fcff0037950 comp rx=0 tx=0).stop 2024-04-27T04:12:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.511+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:12:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.511+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.511+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.511+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.511+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcfe4054300 2024-04-27T04:12:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.511+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ad0 con 0x7fcff8101eb0 2024-04-27T04:12:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.512+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:12:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.512+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:12:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.512+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.512+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052c10 con 0x7fcfe4054300 2024-04-27T04:12:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.513+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400ebe0 tx=0x7fcff00487b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:12:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.514+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4054300 2024-04-27T04:12:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.514+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4054300 2024-04-27T04:12:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:48.515+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe802fb30 con 0x7fcfe4054300 2024-04-27T04:12:48.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:12:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:12:48.512+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 3638 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:12:48.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:48.573+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:48.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:48.573+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:48.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:48.573+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:48.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:48.574+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:48.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:48.353+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:48.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:48.353+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:48.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:48.353+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:48.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:48.353+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:49.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:48.863+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:49.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:48.863+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:49.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:48.863+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:49.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:48.864+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:49.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:49.546+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:49.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:49.546+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:49.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:49.546+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:49.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:49.547+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:49.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:49.351+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:49.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:49.351+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:49.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:49.351+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:49.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:49.353+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:50.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:49.904+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:50.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:49.904+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:50.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:49.904+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:50.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:49.905+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:50.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:50.386+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:50.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:50.386+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:50.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:50.386+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:50.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:50.387+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:50.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:50.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:50.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:50.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:50.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:50.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:50.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:50.595+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:51.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:50.913+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:51.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:50.913+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:51.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:50.913+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:51.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:50.913+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:51.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:51.395+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:51.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:51.395+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:51.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:51.395+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:51.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:51.396+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:51.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:51.631+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:51.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:51.631+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:51.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:51.631+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:51.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:51.632+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:52.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:51.911+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:52.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:51.911+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:52.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:51.911+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:52.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:51.911+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:52.894 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:52.423+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:52.894 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:52.423+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:52.894 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:52.423+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:52.894 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:52.424+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:52.895 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:52.670+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:52.895 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:52.670+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:52.895 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:52.670+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:52.895 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:52.671+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:53.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:52.895+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:53.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:52.895+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:53.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:52.895+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:53.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:52.896+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:53.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.512+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:12:53.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.512+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:12:53.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.512+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400ebe0 tx=0x7fcff00487b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:12:53.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.512+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400ebe0 tx=0x7fcff00487b0 comp rx=0 tx=0).stop 2024-04-27T04:12:53.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.512+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:12:53.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.512+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:53.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.512+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:53.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.512+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:53.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.512+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcfe4044cf0 2024-04-27T04:12:53.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.512+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ad0 con 0x7fcff8101eb0 2024-04-27T04:12:53.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.513+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:12:53.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.513+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:12:53.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.513+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:53.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.513+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052c10 con 0x7fcfe4044cf0 2024-04-27T04:12:53.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.514+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804da20 tx=0x7fcfe804fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:12:53.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.514+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4044cf0 2024-04-27T04:12:53.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.514+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4044cf0 2024-04-27T04:12:53.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.515+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe802fb30 con 0x7fcfe4044cf0 2024-04-27T04:12:53.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.525+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:12:53.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:53.525+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:12:53.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:12:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:12:53.513+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 3700 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:12:53.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:53.420+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:53.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:53.420+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:53.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:53.420+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:53.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:53.420+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:53.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:53.660+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:53.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:53.660+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:53.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:53.660+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:53.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:53.661+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:54.419 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:53.935+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:54.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:53.936+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:54.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:53.936+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:54.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:53.936+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:54.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:54.420+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:54.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:54.421+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:54.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:54.421+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:54.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:54.421+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:55.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:54.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:55.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:54.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:55.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:54.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:55.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:54.705+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:54.924+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:54.924+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:54.924+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:54.924+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:55.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:55.372+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:55.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:55.372+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:55.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:55.372+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:55.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:55.372+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:56.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:55.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:56.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:55.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:56.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:55.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:56.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:55.722+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:56.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:55.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:56.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:55.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:56.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:55.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:56.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:55.961+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:56.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:56.401+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:56.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:56.401+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:56.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:56.401+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:56.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:56.402+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:57.090 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:56.913+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:57.091 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:56.913+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:57.091 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:56.913+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:57.091 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:56.914+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:57.091 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:56.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:57.091 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:56.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:57.091 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:56.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:57.091 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:56.697+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:57.719 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:57.434+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:57.719 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:57.434+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:57.719 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:57.434+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:57.719 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:57.435+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:58.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:57.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:58.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:57.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:58.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:57.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:58.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:57.721+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:58.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:57.896+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:58.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:57.896+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:58.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:57.896+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:58.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:57.897+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:58.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.513+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:12:58.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.513+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:12:58.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.513+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804da20 tx=0x7fcfe804fa10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:12:58.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.513+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804da20 tx=0x7fcfe804fa10 comp rx=0 tx=0).stop 2024-04-27T04:12:58.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.513+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:12:58.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.513+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:58.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.513+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:58.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.513+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:12:58.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.513+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052af0 con 0x7fcfe4054300 2024-04-27T04:12:58.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.513+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c30 con 0x7fcff8101eb0 2024-04-27T04:12:58.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.514+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:12:58.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.515+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:12:58.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.515+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:12:58.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.515+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052d70 con 0x7fcfe4054300 2024-04-27T04:12:58.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.515+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80424d0 tx=0x7fcfe800b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:12:58.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.516+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4054300 2024-04-27T04:12:58.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.517+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4054300 2024-04-27T04:12:58.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:12:58.518+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe802fb30 con 0x7fcfe4054300 2024-04-27T04:12:58.873 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:12:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:12:58.513+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 3763 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:12:58.873 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:58.479+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:58.873 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:58.479+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:58.873 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:58.479+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:58.873 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:58.480+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:58.874 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:58.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:58.874 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:58.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:58.874 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:58.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:58.874 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:58.679+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:59.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:58.874+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:59.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:58.874+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:59.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:58.874+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:12:59.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:58.875+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:12:59.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:59.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:59.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:59.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:59.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:59.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:12:59.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:12:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:12:59.707+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:12:59.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:59.499+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:59.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:59.499+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:59.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:59.499+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:12:59.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:12:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:12:59.500+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:00.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:59.923+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:00.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:59.923+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:00.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:59.923+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:00.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:12:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:12:59.924+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:00.712 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:00.457+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:00.712 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:00.457+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:00.712 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:00.457+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:00.712 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:00.458+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:01.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:00.713+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:01.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:00.713+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:01.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:00.713+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:01.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:00.714+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:01.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:00.948+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:01.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:00.948+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:01.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:00.948+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:01.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:00.949+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:01.904 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:01.448+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:01.904 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:01.448+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:01.905 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:01.448+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:01.905 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:01.449+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:01.905 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:01.664+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:01.905 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:01.664+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:01.905 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:01.664+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:01.905 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:01.665+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:02.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:01.905+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:02.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:01.905+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:02.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:01.905+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:02.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:01.905+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:02.741 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:02.447+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:02.741 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:02.447+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:02.741 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:02.447+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:02.741 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:02.448+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:02.741 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:02.685+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:02.741 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:02.685+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:02.741 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:02.685+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:02.741 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:02.686+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:03.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:02.903+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:03.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:02.903+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:03.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:02.903+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:03.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:02.904+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:03.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.514+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:13:03.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.514+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:13:03.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.514+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80424d0 tx=0x7fcfe800b6c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:13:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.514+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80424d0 tx=0x7fcfe800b6c0 comp rx=0 tx=0).stop 2024-04-27T04:13:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.514+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:13:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.514+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.514+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.514+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.514+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052af0 con 0x7fcfe4044cf0 2024-04-27T04:13:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.514+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c30 con 0x7fcff8101eb0 2024-04-27T04:13:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.514+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:13:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.515+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:13:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.515+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.515+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052d70 con 0x7fcfe4044cf0 2024-04-27T04:13:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.515+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fcff0010830 tx=0x7fcfe8054cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:13:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.516+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003a340 con 0x7fcfe4044cf0 2024-04-27T04:13:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.516+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0009400 con 0x7fcfe4044cf0 2024-04-27T04:13:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:03.517+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059040 con 0x7fcfe4044cf0 2024-04-27T04:13:03.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:13:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:13:03.514+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 3824 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:13:03.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:03.682+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:03.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:03.682+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:03.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:03.682+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:03.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:03.683+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:03.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:03.462+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:03.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:03.462+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:03.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:03.462+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:03.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:03.463+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:04.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:03.950+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:04.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:03.950+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:04.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:03.950+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:04.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:03.951+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:04.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:04.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:04.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:04.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:04.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:04.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:04.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:04.699+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:04.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:04.454+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:04.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:04.454+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:04.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:04.454+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:04.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:04.455+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:05.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:04.933+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:05.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:04.933+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:05.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:04.933+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:05.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:04.933+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:05.813 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:05.495+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:05.813 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:05.496+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:05.813 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:05.496+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:05.813 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:05.497+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:05.814 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:05.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:05.814 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:05.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:05.814 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:05.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:05.814 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:05.713+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:06.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:05.981+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:06.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:05.981+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:06.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:05.981+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:06.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:05.982+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:06.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:06.536+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:06.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:06.536+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:06.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:06.536+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:06.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:06.538+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:06.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:06.710+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:06.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:06.710+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:06.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:06.710+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:06.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:06.711+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:07.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:07.024+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:07.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:07.024+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:07.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:07.024+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:07.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:07.025+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:07.760 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:07.499+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:07.760 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:07.499+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:07.760 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:07.499+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:07.760 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:07.500+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:07.760 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:07.761+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:07.760 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:07.761+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:07.760 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:07.761+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:08.010 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:07.762+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:08.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:08.011+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:08.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:08.011+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:08.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:08.011+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:08.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:08.012+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:08.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.514+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:13:08.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.514+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:13:08.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.514+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fcff0010830 tx=0x7fcfe8054cc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:13:08.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.514+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fcff0010830 tx=0x7fcfe8054cc0 comp rx=0 tx=0).stop 2024-04-27T04:13:08.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.514+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:13:08.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.514+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:08.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.514+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:08.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.515+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:08.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.515+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052af0 con 0x7fcfe4054300 2024-04-27T04:13:08.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.515+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c30 con 0x7fcff8101eb0 2024-04-27T04:13:08.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.515+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:13:08.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.516+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:13:08.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.516+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:08.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.516+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052d70 con 0x7fcfe4054300 2024-04-27T04:13:08.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.516+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fcfe404f5b0 tx=0x7fcff00193d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:13:08.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.516+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4054300 2024-04-27T04:13:08.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.516+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4054300 2024-04-27T04:13:08.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.517+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe80373a0 con 0x7fcfe4054300 2024-04-27T04:13:08.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.885+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:13:08.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:08.885+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:13:08.884 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:13:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:13:08.515+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 3886 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:13:08.884 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:08.502+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:08.884 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:08.502+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:08.885 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:08.502+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:08.885 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:08.503+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:08.885 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:08.750+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:08.885 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:08.750+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:08.885 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:08.750+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:08.885 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:08.750+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:09.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:09.005+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:09.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:09.005+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:09.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:09.005+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:09.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:09.006+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:09.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:09.517+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:09.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:09.517+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:09.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:09.517+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:09.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:09.517+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:09.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:09.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:09.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:09.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:09.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:09.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:09.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:09.704+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:10.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:10.006+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:10.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:10.006+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:10.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:10.006+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:10.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:10.006+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:10.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:10.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:10.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:10.490+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:10.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:10.490+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:10.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:10.490+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:10.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:10.664+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:10.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:10.664+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:10.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:10.664+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:10.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:10.665+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:11.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:10.998+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:11.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:10.998+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:11.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:10.998+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:11.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:10.998+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:11.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:11.631+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:11.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:11.631+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:11.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:11.631+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:11.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:11.632+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:11.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:11.507+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:11.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:11.507+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:11.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:11.507+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:11.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:11.508+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:12.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:12.037+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:12.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:12.037+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:12.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:12.037+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:12.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:12.037+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:12.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:12.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:12.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:12.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:12.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:12.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:12.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:12.667+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:12.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:12.550+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:12.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:12.550+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:12.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:12.550+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:12.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:12.550+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:13.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:13.083+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:13.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:13.083+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:13.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:13.083+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:13.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:13.084+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:13.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.515+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:13:13.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.515+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:13:13.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.515+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fcfe404f5b0 tx=0x7fcff00193d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:13:13.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.515+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fcfe404f5b0 tx=0x7fcff00193d0 comp rx=0 tx=0).stop 2024-04-27T04:13:13.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.516+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:13:13.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.516+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:13.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.516+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:13.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.516+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:13.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.516+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b80 con 0x7fcfe4044cf0 2024-04-27T04:13:13.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.516+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052cc0 con 0x7fcff8101eb0 2024-04-27T04:13:13.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.516+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:13:13.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.517+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:13:13.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.517+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:13.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.517+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e00 con 0x7fcfe4044cf0 2024-04-27T04:13:13.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.518+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fcff0037920 tx=0x7fcff0003ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:13:13.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.519+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8037930 con 0x7fcfe4044cf0 2024-04-27T04:13:13.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.519+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4044cf0 2024-04-27T04:13:13.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:13.521+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe80373a0 con 0x7fcfe4044cf0 2024-04-27T04:13:13.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:13:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:13:13.516+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 3948 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:13:13.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:13.676+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:13.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:13.676+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:13.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:13.676+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:13.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:13.678+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:13.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:13.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:13.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:13.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:13.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:13.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:13.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:13.501+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:14.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:14.103+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:14.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:14.103+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:14.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:14.103+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:14.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:14.104+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:14.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:14.512+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:14.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:14.512+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:14.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:14.512+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:14.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:14.513+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:14.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:14.728+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:14.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:14.728+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:14.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:14.728+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:14.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:14.729+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:15.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:15.088+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:15.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:15.088+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:15.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:15.088+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:15.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:15.090+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:15.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:15.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:15.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:15.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:15.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:15.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:15.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:15.707+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:15.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:15.557+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:15.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:15.557+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:15.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:15.557+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:15.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:15.558+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:16.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:16.138+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:16.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:16.138+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:16.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:16.138+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:16.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:16.138+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:16.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:16.699+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:16.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:16.699+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:16.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:16.699+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:16.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:16.700+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:16.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:16.579+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:16.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:16.579+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:16.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:16.579+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:16.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:16.579+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:17.109+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:17.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:17.109+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:17.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:17.109+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:17.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:17.110+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:17.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:17.582+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:17.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:17.582+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:17.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:17.582+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:17.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:17.583+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:17.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:17.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:17.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:17.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:17.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:17.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:17.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:17.694+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:18.146+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:18.146+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:18.146+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:18.147+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:18.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.516+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:13:18.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.516+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:13:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.516+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fcff0037920 tx=0x7fcff0003ec0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:13:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.516+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fcff0037920 tx=0x7fcff0003ec0 comp rx=0 tx=0).stop 2024-04-27T04:13:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.516+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:13:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.516+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.516+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.516+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.516+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b80 con 0x7fcfe4054300 2024-04-27T04:13:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.516+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052cc0 con 0x7fcff8101eb0 2024-04-27T04:13:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.517+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:13:18.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.517+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:13:18.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.517+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:18.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.517+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e00 con 0x7fcfe4054300 2024-04-27T04:13:18.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.518+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fcff0052050 tx=0x7fcfe80537c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:13:18.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.519+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003a340 con 0x7fcfe4054300 2024-04-27T04:13:18.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.520+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0009400 con 0x7fcfe4054300 2024-04-27T04:13:18.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:18.520+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059040 con 0x7fcfe4054300 2024-04-27T04:13:18.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:13:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:13:18.517+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 4011 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:13:18.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:18.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:18.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:18.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:18.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:18.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:18.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:18.679+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:18.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:18.620+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:18.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:18.620+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:18.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:18.620+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:18.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:18.621+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:19.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:19.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:19.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:19.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:19.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:19.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:19.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:19.177+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:19.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:19.632+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:19.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:19.632+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:19.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:19.632+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:19.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:19.633+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:19.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:19.715+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:19.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:19.715+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:19.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:19.715+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:19.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:19.716+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:20.129+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:20.129+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:20.129+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:20.130+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:20.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:20.714+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:20.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:20.714+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:20.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:20.714+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:20.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:20.714+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:20.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:20.642+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:20.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:20.642+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:20.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:20.642+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:20.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:20.643+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:21.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:21.109+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:21.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:21.109+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:21.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:21.109+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:21.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:21.110+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:21.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:21.636+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:21.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:21.636+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:21.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:21.636+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:21.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:21.638+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:21.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:21.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:21.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:21.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:21.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:21.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:21.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:21.735+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:22.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:22.137+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:22.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:22.137+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:22.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:22.137+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:22.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:22.138+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:22.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:22.600+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:22.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:22.600+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:22.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:22.600+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:22.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:22.601+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:22.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:22.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:22.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:22.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:22.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:22.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:22.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:22.721+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:23.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:23.137+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:23.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:23.137+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:23.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:23.137+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:23.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:23.138+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:23.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.517+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:13:23.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.517+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:13:23.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.517+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fcff0052050 tx=0x7fcfe80537c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:13:23.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.517+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fcff0052050 tx=0x7fcfe80537c0 comp rx=0 tx=0).stop 2024-04-27T04:13:23.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.517+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:13:23.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.517+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:23.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.517+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:23.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.517+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:23.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.517+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b80 con 0x7fcfe4044cf0 2024-04-27T04:13:23.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.517+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052cc0 con 0x7fcff8101eb0 2024-04-27T04:13:23.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.518+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:13:23.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.518+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:13:23.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.518+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:23.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.518+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e00 con 0x7fcfe4044cf0 2024-04-27T04:13:23.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.519+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4043e00 tx=0x7fcff000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:13:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.520+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8037930 con 0x7fcfe4044cf0 2024-04-27T04:13:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.520+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8054cc0 con 0x7fcfe4044cf0 2024-04-27T04:13:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:23.521+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe804d620 con 0x7fcfe4044cf0 2024-04-27T04:13:23.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:13:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:13:23.517+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 4072 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:13:23.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:23.675+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:23.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:23.675+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:23.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:23.675+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:23.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:23.676+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:23.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:23.625+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:23.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:23.625+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:23.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:23.625+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:23.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:23.626+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:24.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:24.245+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:13:24.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:24.245+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:13:24.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:24.095+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:24.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:24.095+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:24.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:24.095+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:24.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:24.095+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:24.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:24.685+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:24.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:24.685+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:24.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:24.685+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:24.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:24.686+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:24.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:24.601+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:24.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:24.601+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:24.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:24.601+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:24.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:24.602+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:25.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:25.094+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:25.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:25.094+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:25.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:25.094+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:25.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:25.095+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:25.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:25.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:25.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:25.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:25.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:25.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:25.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:25.668+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:25.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:25.592+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:25.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:25.592+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:25.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:25.592+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:25.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:25.593+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:26.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:26.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:26.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:26.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:26.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:26.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:26.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:26.070+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:26.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:26.715+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:26.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:26.715+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:26.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:26.715+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:26.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:26.716+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:26.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:26.625+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:26.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:26.625+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:26.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:26.625+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:26.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:26.626+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:27.410 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:27.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:27.410 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:27.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:27.410 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:27.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:27.410 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:27.074+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:27.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:27.671+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:27.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:27.671+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:27.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:27.671+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:27.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:27.672+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:27.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:27.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:27.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:27.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:27.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:27.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:27.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:27.761+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:28.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:28.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:28.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:28.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:28.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:28.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:28.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:28.079+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:28.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.517+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:13:28.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.517+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:13:28.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.517+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4043e00 tx=0x7fcff000edf0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:13:28.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.517+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4043e00 tx=0x7fcff000edf0 comp rx=0 tx=0).stop 2024-04-27T04:13:28.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.517+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:13:28.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.517+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:28.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.518+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:28.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.518+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:28.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.518+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b80 con 0x7fcfe4054300 2024-04-27T04:13:28.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.518+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052cc0 con 0x7fcff8101eb0 2024-04-27T04:13:28.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.518+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:13:28.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.518+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:13:28.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.518+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:28.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.518+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e00 con 0x7fcfe4054300 2024-04-27T04:13:28.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.519+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fcff0037920 tx=0x7fcfe800b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:13:28.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.520+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0009e80 con 0x7fcfe4054300 2024-04-27T04:13:28.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.520+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0003ae0 con 0x7fcfe4054300 2024-04-27T04:13:28.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:28.521+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4054300 2024-04-27T04:13:28.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:13:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:13:28.518+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 4134 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:13:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:28.706+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:28.706+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:28.706+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:28.706+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:28.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:28.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:28.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:28.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:28.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:28.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:28.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:28.758+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:29.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:29.115+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:29.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:29.115+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:29.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:29.115+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:29.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:29.116+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:29.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:29.659+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:29.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:29.659+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:29.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:29.659+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:29.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:29.660+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:29.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:29.733+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:29.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:29.733+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:29.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:29.733+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:29.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:29.734+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:30.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:30.088+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:30.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:30.088+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:30.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:30.088+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:30.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:30.089+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:31.071 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:30.690+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:31.071 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:30.690+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:31.071 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:30.690+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:31.071 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:30.691+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:31.071 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:30.771+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:31.071 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:30.771+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:31.071 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:30.771+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:31.072 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:30.771+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:31.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:31.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:31.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:31.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:31.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:31.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:31.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:31.072+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:31.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:31.745+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:31.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:31.745+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:31.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:31.745+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:31.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:31.746+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:31.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:31.705+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:31.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:31.705+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:31.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:31.705+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:31.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:31.706+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:32.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:32.104+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:32.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:32.104+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:32.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:32.104+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:32.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:32.105+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:32.718 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:32.661+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:32.719 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:32.661+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:32.719 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:32.661+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:32.719 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:32.662+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:33.148 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:32.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:33.148 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:32.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:33.148 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:32.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:33.148 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:32.721+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:33.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:33.149+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:33.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:33.149+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:33.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:33.149+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:33.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:33.150+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.519+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:13:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.519+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:13:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.519+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fcff0037920 tx=0x7fcfe800b940 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:13:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.519+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fcff0037920 tx=0x7fcfe800b940 comp rx=0 tx=0).stop 2024-04-27T04:13:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.519+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:13:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.519+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.519+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.519+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.519+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b80 con 0x7fcfe4044cf0 2024-04-27T04:13:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.519+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052cc0 con 0x7fcff8101eb0 2024-04-27T04:13:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.519+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:13:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.520+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:13:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.520+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.520+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e00 con 0x7fcfe4044cf0 2024-04-27T04:13:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.520+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8028f90 tx=0x7fcff0003ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:13:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.521+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe80465e0 con 0x7fcfe4044cf0 2024-04-27T04:13:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.521+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4044cf0 2024-04-27T04:13:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:33.522+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe804d620 con 0x7fcfe4044cf0 2024-04-27T04:13:33.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:13:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:13:33.519+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 4196 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:13:33.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:33.719+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:33.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:33.719+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:33.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:33.719+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:33.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:33.719+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:33.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:33.709+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:33.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:33.709+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:33.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:33.709+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:33.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:33.710+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:34.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:34.125+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:34.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:34.125+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:34.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:34.125+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:34.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:34.126+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:35.140 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:34.724+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:35.141 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:34.724+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:35.141 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:34.724+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:35.141 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:34.724+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:35.141 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:34.766+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:35.141 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:34.766+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:35.141 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:34.766+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:35.141 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:34.767+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:35.141+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:35.141+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:35.141+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:35.142+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:36.161 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:35.795+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:36.161 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:35.795+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:36.161 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:35.795+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:36.161 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:35.796+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:36.162 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:35.738+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:36.162 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:35.738+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:36.162 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:35.738+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:36.162 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:35.739+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:36.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:36.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:36.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:36.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:36.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:36.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:36.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:36.162+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:36.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:36.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:36.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:36.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:36.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:36.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:36.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:36.776+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:36.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:36.754+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:36.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:36.754+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:36.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:36.754+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:36.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:36.754+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:37.518 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:37.192+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:37.518 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:37.192+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:37.518 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:37.192+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:37.518 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:37.192+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:37.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:37.765+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:37.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:37.765+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:37.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:37.765+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:37.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:37.766+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:37.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:37.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:37.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:37.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:37.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:37.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:37.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:37.750+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:38.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.520+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:13:38.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.520+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:13:38.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.520+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8028f90 tx=0x7fcff0003ec0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:13:38.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.520+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8028f90 tx=0x7fcff0003ec0 comp rx=0 tx=0).stop 2024-04-27T04:13:38.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.520+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:13:38.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.520+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:38.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.520+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:38.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.520+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:38.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.520+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c30 con 0x7fcfe4054300 2024-04-27T04:13:38.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.520+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052d70 con 0x7fcff8101eb0 2024-04-27T04:13:38.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.520+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:13:38.520 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:38.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:38.520 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:38.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:38.520 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:38.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:38.520 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:38.184+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.521+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:13:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.521+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.521+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052eb0 con 0x7fcfe4054300 2024-04-27T04:13:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.522+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fcfe402a320 tx=0x7fcfe800b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:13:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.522+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0009e80 con 0x7fcfe4054300 2024-04-27T04:13:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.523+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0037170 con 0x7fcfe4054300 2024-04-27T04:13:38.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:38.523+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4054300 2024-04-27T04:13:38.804 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:13:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:13:38.520+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 4259 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:13:38.804 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:38.739+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:38.805 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:38.739+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:38.805 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:38.739+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:38.805 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:38.740+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:39.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:38.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:39.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:38.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:39.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:38.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:39.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:38.806+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:39.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:39.605+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:13:39.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:39.605+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:13:39.604 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:39.213+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:39.604 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:39.213+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:39.604 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:39.213+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:39.605 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:39.213+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:39.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:39.779+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:39.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:39.779+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:39.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:39.779+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:39.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:39.780+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:39.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:39.725+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:39.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:39.725+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:39.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:39.725+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:39.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:39.726+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:40.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:40.167+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:40.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:40.167+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:40.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:40.167+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:40.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:40.168+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:41.120 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:40.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:41.121 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:40.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:41.121 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:40.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:41.121 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:40.824+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:41.121 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:40.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:41.122 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:40.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:41.122 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:40.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:41.122 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:40.679+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:41.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:41.121+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:41.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:41.121+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:41.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:41.121+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:41.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:41.122+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:41.866 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:41.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:41.867 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:41.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:41.867 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:41.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:41.867 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:41.718+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:42.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:41.868+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:42.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:41.868+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:42.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:41.868+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:42.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:41.868+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:42.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:42.085+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:42.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:42.085+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:42.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:42.085+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:42.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:42.085+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:42.844 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:42.727+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:42.844 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:42.727+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:42.845 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:42.727+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:42.845 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:42.728+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:43.117 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:42.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:43.118 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:42.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:43.118 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:42.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:43.118 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:42.846+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:43.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:43.119+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:43.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:43.119+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:43.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:43.119+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:43.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:43.120+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:43.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.520+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:13:43.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.520+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:13:43.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.521+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fcfe402a320 tx=0x7fcfe800b940 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:13:43.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.521+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fcfe402a320 tx=0x7fcfe800b940 comp rx=0 tx=0).stop 2024-04-27T04:13:43.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.521+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:13:43.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.521+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:43.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.521+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:43.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.521+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:43.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.521+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c30 con 0x7fcfe4044cf0 2024-04-27T04:13:43.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.521+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052d70 con 0x7fcff8101eb0 2024-04-27T04:13:43.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.522+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:13:43.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.522+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:13:43.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.522+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:43.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.522+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052eb0 con 0x7fcfe4044cf0 2024-04-27T04:13:43.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.523+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fcff0052050 tx=0x7fcff003eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:13:43.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.523+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0009e80 con 0x7fcfe4044cf0 2024-04-27T04:13:43.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.523+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff002baf0 con 0x7fcfe4044cf0 2024-04-27T04:13:43.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:43.525+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4044cf0 2024-04-27T04:13:43.865 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:13:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:13:43.521+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 4320 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:13:43.865 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:43.767+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:43.865 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:43.767+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:43.865 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:43.767+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:43.865 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:43.768+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:44.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:44.091+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:44.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:44.091+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:44.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:44.091+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:44.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:44.092+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:44.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:43.866+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:44.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:43.866+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:44.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:43.866+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:44.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:43.867+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:45.065 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:44.880+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:45.065 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:44.880+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:45.065 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:44.880+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:45.065 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:44.881+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:45.065 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:44.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:45.065 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:44.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:45.065 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:44.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:45.065 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:44.735+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:45.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:45.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:45.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:45.067+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:46.112 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:45.780+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:46.112 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:45.780+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:46.112 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:45.780+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:46.112 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:45.780+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:46.112 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:45.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:46.113 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:45.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:46.113 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:45.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:46.113 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:45.847+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:46.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:46.113+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:46.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:46.113+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:46.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:46.113+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:46.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:46.114+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:47.121 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:46.822+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:47.121 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:46.822+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:47.121 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:46.822+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:47.121 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:46.824+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:47.121 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:46.822+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:47.121 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:46.822+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:47.122 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:46.822+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:47.122 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:46.823+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:47.122+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:47.122+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:47.122+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:47.122+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:47.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:47.775+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:47.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:47.775+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:47.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:47.775+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:47.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:47.775+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:47.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:47.842+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:47.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:47.842+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:47.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:47.842+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:47.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:47.843+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:48.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:48.094+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:48.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:48.094+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:48.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:48.094+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:48.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:48.095+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:48.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.522+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:13:48.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.522+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:13:48.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.522+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fcff0052050 tx=0x7fcff003eb60 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:13:48.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.522+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fcff0052050 tx=0x7fcff003eb60 comp rx=0 tx=0).stop 2024-04-27T04:13:48.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.522+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:13:48.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.522+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:48.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.522+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.522+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.522+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b80 con 0x7fcfe4054300 2024-04-27T04:13:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.522+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052cc0 con 0x7fcff8101eb0 2024-04-27T04:13:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.522+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:13:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.523+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:13:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.523+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.523+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e00 con 0x7fcfe4054300 2024-04-27T04:13:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.523+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803cc80 tx=0x7fcff000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:13:48.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.525+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4054300 2024-04-27T04:13:48.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.525+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8042ed0 con 0x7fcfe4054300 2024-04-27T04:13:48.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:48.526+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013a90 con 0x7fcfe4054300 2024-04-27T04:13:48.820 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:13:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:13:48.522+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 4382 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:13:48.820 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:48.768+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:48.821 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:48.768+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:48.821 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:48.768+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:48.821 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:48.768+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:49.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:49.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:49.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:49.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:49.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:49.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:49.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:49.068+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:49.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:48.847+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:49.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:48.847+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:49.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:48.847+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:49.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:48.848+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:50.035 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:49.896+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:50.035 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:49.896+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:50.035 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:49.896+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:50.035 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:49.896+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:50.035 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:49.785+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:50.035 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:49.785+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:50.036 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:49.785+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:50.036 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:49.786+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:50.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:50.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:50.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:50.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:50.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:50.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:50.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:50.037+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:51.004 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:50.754+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:51.004 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:50.754+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:51.004 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:50.754+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:51.005 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:50.755+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:51.005 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:50.850+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:51.005 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:50.850+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:51.005 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:50.850+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:51.005 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:50.851+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:51.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:51.005+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:51.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:51.005+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:51.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:51.005+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:51.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:51.006+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:51.892 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:51.760+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:51.892 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:51.760+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:51.893 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:51.760+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:51.893 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:51.761+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:51.893 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:51.801+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:51.893 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:51.801+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:51.893 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:51.801+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:51.893 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:51.802+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:52.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:52.019+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:52.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:52.019+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:52.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:52.019+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:52.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:52.019+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:53.034 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:52.780+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:53.035 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:52.780+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:53.035 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:52.780+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:53.035 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:52.781+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:53.035 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:52.782+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:53.035 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:52.782+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:53.035 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:52.782+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:53.035 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:52.783+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:53.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:53.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:53.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:53.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:53.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:53.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:53.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:53.036+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:53.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.522+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:13:53.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.523+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:13:53.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.523+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803cc80 tx=0x7fcff000edf0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:13:53.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.523+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803cc80 tx=0x7fcff000edf0 comp rx=0 tx=0).stop 2024-04-27T04:13:53.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.523+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:13:53.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.523+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:53.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.523+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:53.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.523+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:53.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.523+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b80 con 0x7fcfe4044cf0 2024-04-27T04:13:53.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.523+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052cc0 con 0x7fcff8101eb0 2024-04-27T04:13:53.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.523+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:13:53.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.524+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:13:53.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.524+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:53.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.524+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e00 con 0x7fcfe4044cf0 2024-04-27T04:13:53.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.524+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fcff002a210 tx=0x7fcfe800b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:13:53.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.526+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0009e80 con 0x7fcfe4044cf0 2024-04-27T04:13:53.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.526+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff003ee20 con 0x7fcfe4044cf0 2024-04-27T04:13:53.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:53.527+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059040 con 0x7fcfe4044cf0 2024-04-27T04:13:53.772 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:13:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:13:53.523+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 4444 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:13:53.773 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:53.773+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:53.773 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:53.773+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:53.773 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:53.773+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:53.773 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:53.774+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:53.774 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:53.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:53.774 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:53.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:53.774 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:53.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:53.774 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:53.748+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:54.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:54.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:54.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:54.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:54.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:54.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:54.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:54.050+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:54.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:54.965+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:13:54.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:54.965+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:13:55.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:54.783+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:55.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:54.783+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:55.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:54.783+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:55.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:54.784+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:55.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:55.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:55.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:55.002+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:55.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:54.754+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:55.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:54.754+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:55.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:54.754+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:55.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:54.754+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:55.985 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:55.831+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:55.985 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:55.831+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:55.986 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:55.831+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:55.986 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:55.832+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:55.986 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:55.731+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:55.986 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:55.731+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:55.986 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:55.731+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:55.986 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:55.732+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:56.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:55.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:56.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:55.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:56.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:55.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:56.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:55.987+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:56.970 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:56.688+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:56.971 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:56.688+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:56.971 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:56.688+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:56.971 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:56.689+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:56.971 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:56.830+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:56.971 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:56.830+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:56.971 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:56.830+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:56.971 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:56.831+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:57.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:56.971+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:57.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:56.971+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:57.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:56.971+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:57.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:56.972+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:58.036 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:57.928+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:58.037 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:57.928+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:58.037 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:57.928+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:58.037 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:57.929+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:58.037 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:57.715+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:58.037 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:57.715+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:58.037 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:57.715+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:58.037 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:57.715+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:58.037 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:57.784+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:58.038 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:57.784+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:58.038 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:57.784+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:58.038 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:57.785+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:58.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.524+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:13:58.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.524+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:13:58.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.524+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fcff002a210 tx=0x7fcfe800b6c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:13:58.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.524+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fcff002a210 tx=0x7fcfe800b6c0 comp rx=0 tx=0).stop 2024-04-27T04:13:58.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.524+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:13:58.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.524+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:58.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.524+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:58.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.524+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:13:58.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.524+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b80 con 0x7fcfe4054300 2024-04-27T04:13:58.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.524+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052cc0 con 0x7fcff8101eb0 2024-04-27T04:13:58.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.525+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:13:58.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.525+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:13:58.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.525+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:13:58.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.525+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e00 con 0x7fcfe4054300 2024-04-27T04:13:58.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.526+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400f920 tx=0x7fcff0011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:13:58.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.526+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0009e80 con 0x7fcfe4054300 2024-04-27T04:13:58.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.526+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0004000 con 0x7fcfe4054300 2024-04-27T04:13:58.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:13:58.528+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059040 con 0x7fcfe4054300 2024-04-27T04:13:58.882 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:13:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:13:58.524+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 4507 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:13:58.883 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:58.751+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:58.883 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:58.751+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:58.883 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:58.751+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:13:58.883 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:58.752+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:13:58.883 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:58.751+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:58.883 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:58.751+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:58.883 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:58.751+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:13:58.884 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:58.752+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:13:59.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:58.884+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:59.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:58.884+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:59.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:58.884+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:13:59.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:58.884+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:00.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:59.756+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:00.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:59.756+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:00.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:59.756+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:00.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:13:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:13:59.757+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:00.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:59.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:00.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:59.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:00.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:59.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:00.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:13:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:13:59.708+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:00.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:59.846+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:00.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:59.847+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:00.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:59.847+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:00.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:13:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:13:59.847+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:01.072 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:00.884+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:01.072 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:00.884+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:01.072 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:00.884+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:01.072 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:00.885+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:01.073 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:00.754+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:01.073 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:00.754+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:01.073 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:00.754+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:01.073 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:00.756+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:01.073 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:00.674+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:01.073 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:00.674+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:01.073 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:00.674+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:01.074 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:00.675+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:01.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:01.790+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:01.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:01.790+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:01.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:01.790+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:01.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:01.792+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:01.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:01.680+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:01.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:01.680+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:01.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:01.680+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:01.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:01.681+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:02.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:01.925+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:02.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:01.925+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:02.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:01.925+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:02.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:01.926+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:02.958 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:02.806+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:02.959 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:02.806+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:02.959 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:02.806+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:02.959 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:02.807+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:02.959 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:02.705+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:02.959 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:02.705+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:02.959 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:02.705+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:02.959 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:02.706+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:03.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:02.959+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:03.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:02.959+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:03.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:02.959+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:03.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:02.960+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.524+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:14:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.524+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:14:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.524+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400f920 tx=0x7fcff0011c90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:14:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.524+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400f920 tx=0x7fcff0011c90 comp rx=0 tx=0).stop 2024-04-27T04:14:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.524+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:14:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.525+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.525+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.525+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.525+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b80 con 0x7fcfe4044cf0 2024-04-27T04:14:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.525+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052cc0 con 0x7fcff8101eb0 2024-04-27T04:14:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.525+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:14:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.525+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:14:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.526+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.526+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e00 con 0x7fcfe4044cf0 2024-04-27T04:14:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.526+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fcfe404e900 tx=0x7fcff003eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:14:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.526+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4044cf0 2024-04-27T04:14:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.526+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8042ed0 con 0x7fcfe4044cf0 2024-04-27T04:14:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:03.528+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013510 con 0x7fcfe4044cf0 2024-04-27T04:14:03.820 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:14:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:14:03.525+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 4568 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:14:03.821 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:03.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:03.821 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:03.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:03.821 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:03.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:03.821 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:03.719+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:04.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:03.957+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:04.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:03.957+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:04.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:03.957+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:04.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:03.958+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:04.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:03.821+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:04.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:03.821+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:04.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:03.821+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:04.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:03.822+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:05.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:04.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:05.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:04.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:05.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:04.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:05.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:04.823+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:05.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:04.754+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:05.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:04.754+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:05.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:04.754+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:05.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:04.755+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:05.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:04.911+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:05.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:04.911+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:05.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:04.911+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:05.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:04.912+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:05.783+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:05.783+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:05.783+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:05.784+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:06.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:05.796+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:06.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:05.796+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:06.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:05.796+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:06.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:05.797+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:05.884+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:05.884+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:05.884+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:05.885+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:07.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:06.818+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:07.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:06.819+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:07.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:06.819+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:07.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:06.819+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:07.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:06.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:07.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:06.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:07.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:06.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:07.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:06.775+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:07.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:06.897+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:07.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:06.897+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:07.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:06.897+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:07.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:06.898+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:08.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:07.811+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:08.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:07.811+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:08.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:07.811+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:08.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:07.812+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:08.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:07.811+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:08.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:07.811+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:08.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:07.811+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:08.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:07.812+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:08.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:07.867+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:08.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:07.867+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:08.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:07.867+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:08.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:07.868+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:08.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.524+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:14:08.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.524+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:14:08.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.524+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fcfe404e900 tx=0x7fcff003eaf0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:14:08.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.524+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fcfe404e900 tx=0x7fcff003eaf0 comp rx=0 tx=0).stop 2024-04-27T04:14:08.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.524+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:14:08.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.524+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:08.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.524+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:08.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.524+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:08.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.524+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052af0 con 0x7fcfe4054300 2024-04-27T04:14:08.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.524+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c30 con 0x7fcff8101eb0 2024-04-27T04:14:08.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.525+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:14:08.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.526+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:14:08.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.526+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:08.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.526+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052d70 con 0x7fcfe4054300 2024-04-27T04:14:08.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.526+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fcfe801ba80 tx=0x7fcfe800b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:14:08.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.526+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4054300 2024-04-27T04:14:08.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.526+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8017190 con 0x7fcfe4054300 2024-04-27T04:14:08.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:08.528+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013510 con 0x7fcfe4054300 2024-04-27T04:14:08.828 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:14:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:14:08.525+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 4630 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:14:08.828 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:08.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:08.828 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:08.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:08.828 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:08.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:08.828 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:08.773+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:09.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:08.887+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:09.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:08.887+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:09.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:08.887+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:09.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:08.888+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:09.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:08.828+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:09.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:08.828+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:09.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:08.828+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:09.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:08.829+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:10.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:09.826+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:10.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:09.826+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:10.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:09.826+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:10.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:09.827+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:10.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:09.902+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:10.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:09.902+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:10.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:09.902+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:10.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:09.903+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:10.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:09.811+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:10.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:09.811+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:10.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:09.811+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:10.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:09.812+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:10.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:10.324+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:14:10.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:10.324+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:14:11.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:10.825+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:11.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:10.825+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:11.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:10.825+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:11.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:10.827+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:11.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:10.950+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:11.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:10.950+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:11.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:10.950+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:11.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:10.951+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:11.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:10.790+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:11.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:10.790+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:11.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:10.790+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:11.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:10.791+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:11.848 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:11.803+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:11.848 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:11.803+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:11.848 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:11.803+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:11.848 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:11.804+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:12.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:11.848+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:12.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:11.848+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:12.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:11.848+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:12.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:11.849+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:11.944+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:11.944+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:11.944+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:11.945+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:13.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:12.884+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:13.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:12.884+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:13.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:12.884+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:13.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:12.885+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:13.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:12.792+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:13.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:12.792+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:13.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:12.792+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:13.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:12.794+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:13.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:12.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:13.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:12.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:13.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:12.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:13.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:12.992+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:13.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.525+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:14:13.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.525+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:14:13.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.525+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fcfe801ba80 tx=0x7fcfe800b940 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:14:13.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.525+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fcfe801ba80 tx=0x7fcfe800b940 comp rx=0 tx=0).stop 2024-04-27T04:14:13.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.525+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:14:13.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.526+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:13.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.526+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:13.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.526+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:13.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.526+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b80 con 0x7fcfe4044cf0 2024-04-27T04:14:13.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.526+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052cc0 con 0x7fcff8101eb0 2024-04-27T04:14:13.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.526+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:14:13.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.527+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:14:13.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.527+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:13.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.527+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e00 con 0x7fcfe4044cf0 2024-04-27T04:14:13.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.527+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80137c0 tx=0x7fcfe8014900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:14:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.529+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4044cf0 2024-04-27T04:14:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.529+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8017190 con 0x7fcfe4044cf0 2024-04-27T04:14:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:13.530+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013510 con 0x7fcfe4044cf0 2024-04-27T04:14:13.811 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:14:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:14:13.526+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 4692 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:14:14.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:13.811+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:14.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:13.811+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:14.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:13.811+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:14.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:13.812+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:14.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:13.864+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:14.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:13.864+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:14.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:13.864+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:14.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:13.865+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:14.041+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:14.041+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:14.041+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:14.041+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:15.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:14.808+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:15.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:14.808+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:15.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:14.808+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:15.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:14.810+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:15.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:14.851+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:15.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:14.851+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:15.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:14.851+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:15.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:14.852+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:15.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:15.005+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:15.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:15.005+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:15.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:15.005+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:15.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:15.006+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:16.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:15.855+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:16.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:15.855+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:16.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:15.855+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:16.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:15.856+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:16.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:15.854+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:16.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:15.854+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:16.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:15.854+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:16.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:15.854+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:16.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:15.991+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:16.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:15.991+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:16.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:15.991+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:16.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:15.991+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:17.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:16.999+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:17.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:16.999+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:17.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:16.999+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:17.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:16.999+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:17.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:16.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:17.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:16.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:17.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:16.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:17.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:16.846+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:17.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:16.814+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:17.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:16.814+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:17.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:16.814+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:17.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:16.814+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:18.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:17.847+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:18.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:17.847+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:18.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:17.847+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:18.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:17.848+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:18.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:17.832+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:18.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:17.832+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:18.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:17.832+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:18.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:17.832+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:18.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:17.978+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:18.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:17.978+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:18.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:17.978+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:18.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:17.979+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:18.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.526+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:14:18.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.526+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:14:18.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.526+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80137c0 tx=0x7fcfe8014900 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:14:18.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.526+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80137c0 tx=0x7fcfe8014900 comp rx=0 tx=0).stop 2024-04-27T04:14:18.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.526+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:14:18.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.526+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:18.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.526+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:18.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.526+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:18.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.528+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b80 con 0x7fcfe4054300 2024-04-27T04:14:18.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.528+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052cc0 con 0x7fcff8101eb0 2024-04-27T04:14:18.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.528+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:14:18.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.529+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:14:18.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.529+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:18.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.529+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e00 con 0x7fcfe4054300 2024-04-27T04:14:18.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.529+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400d440 tx=0x7fcfe80137f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:14:18.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.530+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4054300 2024-04-27T04:14:18.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.530+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8017190 con 0x7fcfe4054300 2024-04-27T04:14:18.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:18.531+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013510 con 0x7fcfe4054300 2024-04-27T04:14:18.807 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:14:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:14:18.526+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 4755 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:14:19.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:18.816+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:19.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:18.816+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:19.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:18.816+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:19.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:18.817+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:19.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:18.968+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:19.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:18.968+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:19.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:18.968+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:19.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:18.969+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:19.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:18.809+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:19.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:18.809+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:19.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:18.809+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:19.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:18.810+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:20.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:19.818+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:20.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:19.818+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:20.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:19.818+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:20.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:19.819+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:20.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:19.796+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:20.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:19.796+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:20.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:19.796+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:20.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:19.796+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:20.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:19.941+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:20.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:19.941+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:20.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:19.941+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:20.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:19.942+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:21.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:20.800+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.584379+0000 front 2024-04-27T04:03:47.584323+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:21.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:20.800+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.584423+0000 front 2024-04-27T04:03:47.584301+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:21.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:20.800+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.584335+0000 front 2024-04-27T04:03:47.584356+0000 (oldest deadline 2024-04-27T04:04:12.884361+0000) 2024-04-27T04:14:21.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:20.801+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:21.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:20.826+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:21.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:20.826+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:21.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:20.826+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:21.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:20.827+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:21.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:20.906+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:21.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:20.906+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:21.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:20.906+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:21.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:20.907+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:22.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:21.848+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:22.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:21.822+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:22.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:21.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:22.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:21.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:22.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:21.823+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:22.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:21.947+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:22.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:21.947+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:22.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:21.947+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:22.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:21.948+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:23.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:22.865+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:23.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:22.822+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:23.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:22.822+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:23.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:22.822+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:23.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:22.823+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:23.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:22.972+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:23.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:22.972+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:23.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:22.972+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:23.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:22.973+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.528+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:14:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.528+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:14:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.528+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400d440 tx=0x7fcfe80137f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:14:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.528+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400d440 tx=0x7fcfe80137f0 comp rx=0 tx=0).stop 2024-04-27T04:14:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.528+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:14:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.528+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.528+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.528+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.528+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b80 con 0x7fcfe4044cf0 2024-04-27T04:14:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.528+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052cc0 con 0x7fcff8101eb0 2024-04-27T04:14:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.528+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:14:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.529+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:14:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.529+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.529+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e00 con 0x7fcfe4044cf0 2024-04-27T04:14:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.530+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fcff0014f00 tx=0x7fcfe80533c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:14:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.531+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0009e80 con 0x7fcfe4044cf0 2024-04-27T04:14:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.531+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0009e80 con 0x7fcfe4044cf0 2024-04-27T04:14:23.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:23.532+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4044cf0 2024-04-27T04:14:23.836 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:14:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:14:23.528+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 4816 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:14:24.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:23.886+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:24.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:23.837+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:24.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:23.837+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:24.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:23.837+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:24.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:23.838+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:24.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:23.983+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:24.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:23.983+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:24.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:23.983+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:24.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:23.983+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:25.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:24.904+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:25.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:25.001+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.473455+0000 front 2024-04-27T04:03:47.473432+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:25.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:25.001+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.473445+0000 front 2024-04-27T04:03:47.473417+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:25.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:25.001+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.473511+0000 front 2024-04-27T04:03:47.473476+0000 (oldest deadline 2024-04-27T04:04:10.373404+0000) 2024-04-27T04:14:25.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:25.002+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:25.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:24.873+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:25.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:24.873+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:25.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:24.873+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:25.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:24.874+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:25.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:25.685+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:14:25.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:25.685+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:14:25.971 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:25.898+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:25.972 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:25.920+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:25.972 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:25.920+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:25.972 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:25.920+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:25.972 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:25.921+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:26.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:25.973+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:27.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:26.872+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:27.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:27.018+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:27.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:26.957+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 since back 2024-04-27T04:03:47.699255+0000 front 2024-04-27T04:03:47.699338+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:27.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:26.957+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 since back 2024-04-27T04:03:47.699284+0000 front 2024-04-27T04:03:47.699305+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:27.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:26.957+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 since back 2024-04-27T04:03:47.699366+0000 front 2024-04-27T04:03:47.699238+0000 (oldest deadline 2024-04-27T04:04:12.999227+0000) 2024-04-27T04:14:27.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:26.958+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:28.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:27.956+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:28.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:27.852+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:28.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:28.066+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.529+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:14:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.529+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:14:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.529+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fcff0014f00 tx=0x7fcfe80533c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:14:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.529+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fcff0014f00 tx=0x7fcfe80533c0 comp rx=0 tx=0).stop 2024-04-27T04:14:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.529+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:14:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.529+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.529+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.529+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.529+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b80 con 0x7fcfe4054300 2024-04-27T04:14:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.529+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052cc0 con 0x7fcff8101eb0 2024-04-27T04:14:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.530+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:14:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.530+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:14:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.530+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.530+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e00 con 0x7fcfe4054300 2024-04-27T04:14:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.531+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400fb80 tx=0x7fcff004daf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:14:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.531+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0009e80 con 0x7fcfe4054300 2024-04-27T04:14:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.532+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0009e80 con 0x7fcfe4054300 2024-04-27T04:14:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:28.533+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059050 con 0x7fcfe4054300 2024-04-27T04:14:28.882 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:14:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:14:28.529+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 4878 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:14:29.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:29.098+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:29.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:28.970+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:29.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:28.883+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:30.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:29.871+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:30.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:29.969+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:30.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:30.083+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:31.072 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:30.942+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:31.072 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:30.823+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:31.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:31.132+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:32.167 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:31.977+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:32.167 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:31.821+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:32.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:32.168+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:33.113 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:33.014+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:33.113 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:32.836+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:33.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:33.170+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.530+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:14:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.530+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:14:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.530+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400fb80 tx=0x7fcff004daf0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:14:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.530+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400fb80 tx=0x7fcff004daf0 comp rx=0 tx=0).stop 2024-04-27T04:14:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.530+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:14:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.530+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.530+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.530+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.530+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b80 con 0x7fcfe4044cf0 2024-04-27T04:14:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.530+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052cc0 con 0x7fcff8101eb0 2024-04-27T04:14:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.531+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:14:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.531+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:14:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.531+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.531+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e00 con 0x7fcfe4044cf0 2024-04-27T04:14:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.532+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803d3c0 tx=0x7fcff0011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:14:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.533+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4044cf0 2024-04-27T04:14:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.533+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe80083a0 con 0x7fcfe4044cf0 2024-04-27T04:14:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:33.534+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013510 con 0x7fcfe4044cf0 2024-04-27T04:14:33.843 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:14:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:14:33.530+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 4940 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:14:34.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:34.052+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:34.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:33.844+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:34.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:34.179+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:35.160 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:35.003+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:35.161 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:34.824+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:35.162+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:36.117 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:36.017+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:36.117 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:35.860+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:36.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:36.118+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:37.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:36.897+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:37.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:37.065+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:37.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:37.135+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:37.972 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:37.912+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:38.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:38.088+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:38.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:38.117+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:38.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.531+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:14:38.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.531+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:14:38.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.531+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803d3c0 tx=0x7fcff0011a20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:14:38.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.531+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803d3c0 tx=0x7fcff0011a20 comp rx=0 tx=0).stop 2024-04-27T04:14:38.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.531+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:14:38.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.531+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:38.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.531+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:38.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.531+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:38.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.531+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052af0 con 0x7fcfe4054300 2024-04-27T04:14:38.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.531+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c30 con 0x7fcff8101eb0 2024-04-27T04:14:38.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.531+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:14:38.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.532+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:14:38.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.532+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:38.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.532+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052d70 con 0x7fcfe4054300 2024-04-27T04:14:38.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.533+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fcff001af90 tx=0x7fcfe8043010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:14:38.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.534+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003ee10 con 0x7fcfe4054300 2024-04-27T04:14:38.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.534+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0036260 con 0x7fcfe4054300 2024-04-27T04:14:38.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:38.535+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059050 con 0x7fcfe4054300 2024-04-27T04:14:38.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:14:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:14:38.531+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5004 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:14:39.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:39.099+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:39.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:38.932+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:39.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:39.104+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:40.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:39.908+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:40.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:40.101+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:40.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:40.149+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:41.144 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:40.881+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:41.144 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:41.080+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:41.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:41.145+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:42.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:42.125+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:42.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:42.125+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:42.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:41.882+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:43.167 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:43.085+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:43.167 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:42.848+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:43.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:43.168+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:43.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.532+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:14:43.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.532+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:14:43.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.532+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fcff001af90 tx=0x7fcfe8043010 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:14:43.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.532+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fcff001af90 tx=0x7fcfe8043010 comp rx=0 tx=0).stop 2024-04-27T04:14:43.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.532+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:14:43.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.532+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:43.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.532+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:43.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.532+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:43.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.532+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052a60 con 0x7fcfe4044cf0 2024-04-27T04:14:43.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.532+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ba0 con 0x7fcff8101eb0 2024-04-27T04:14:43.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.532+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:14:43.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.533+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:14:43.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.533+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:43.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.533+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052ce0 con 0x7fcfe4044cf0 2024-04-27T04:14:43.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.534+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804bb50 tx=0x7fcff004dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:14:43.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.535+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003ee80 con 0x7fcfe4044cf0 2024-04-27T04:14:43.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.535+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0009e80 con 0x7fcfe4044cf0 2024-04-27T04:14:43.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:43.536+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4044cf0 2024-04-27T04:14:43.827 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:14:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:14:43.532+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5065 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:14:44.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:44.117+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:14:44.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:44.117+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:14:44.116 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:44.060+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:44.116 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:43.828+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:44.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:44.167+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:45.128 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:44.811+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:45.129 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:45.047+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:45.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:45.130+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:46.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:45.843+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:46.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:45.843+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:46.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:45.843+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:46.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:45.844+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:46.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:46.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:46.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:46.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:46.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:46.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:46.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:46.075+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:46.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:46.172+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:47.162 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:47.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:47.162 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:47.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:47.162 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:47.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:47.162 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:47.053+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:47.163 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:46.819+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:47.163 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:46.819+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:47.163 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:46.819+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:47.163 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:46.819+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:47.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:47.163+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:48.120 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:47.865+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:48.120 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:47.865+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:48.120 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:47.865+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:48.120 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:47.866+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:48.120 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:48.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:48.120 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:48.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:48.121 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:48.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:48.121 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:48.017+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:48.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:48.121+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.533+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:14:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.533+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:14:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.533+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804bb50 tx=0x7fcff004dcd0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:14:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.533+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804bb50 tx=0x7fcff004dcd0 comp rx=0 tx=0).stop 2024-04-27T04:14:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.533+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:14:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.533+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.533+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.533+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:48.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.533+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4050970 con 0x7fcfe4054300 2024-04-27T04:14:48.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.533+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052a60 con 0x7fcff8101eb0 2024-04-27T04:14:48.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.533+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:14:48.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.534+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:14:48.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.534+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:48.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.534+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052ba0 con 0x7fcfe4054300 2024-04-27T04:14:48.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.534+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804d970 tx=0x7fcff0025560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:14:48.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.535+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4054300 2024-04-27T04:14:48.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.535+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4054300 2024-04-27T04:14:48.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:48.536+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe804d620 con 0x7fcfe4054300 2024-04-27T04:14:48.856 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:14:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:14:48.533+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5128 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:14:49.120 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:48.858+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:49.120 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:48.858+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:49.120 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:48.858+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:49.120 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:48.859+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:49.121 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:48.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:49.121 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:48.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:49.121 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:48.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:49.121 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:48.986+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:49.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:49.121+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:50.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:50.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:50.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:50.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:50.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:50.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:50.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:50.101+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:50.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:49.893+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:50.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:49.893+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:50.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:49.893+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:50.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:49.895+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:50.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:50.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:50.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:50.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:50.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:50.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:50.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:50.018+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:51.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:50.911+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:51.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:50.911+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:51.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:50.911+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:51.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:50.913+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:51.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:51.072+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:51.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:51.072+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:51.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:51.072+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:51.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:51.073+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:51.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:51.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:51.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:51.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:51.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:51.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:51.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:51.063+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:52.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:52.084+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:52.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:52.084+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:52.084+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:52.086+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:52.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:51.931+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:52.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:51.931+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:52.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:51.931+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:52.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:51.932+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:52.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:52.045+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:52.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:52.045+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:52.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:52.045+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:52.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:52.045+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:53.343 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:52.936+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:53.343 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:52.936+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:53.343 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:52.936+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:53.344 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:52.937+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:53.344 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:53.054+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:53.344 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:53.054+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:53.344 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:53.054+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:53.344 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:53.055+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:53.344 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:53.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:53.344 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:53.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:53.344 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:53.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:53.345 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:53.062+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:53.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.534+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:14:53.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.534+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:14:53.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.534+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804d970 tx=0x7fcff0025560 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:14:53.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.534+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804d970 tx=0x7fcff0025560 comp rx=0 tx=0).stop 2024-04-27T04:14:53.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.534+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:14:53.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.534+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:53.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.534+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:53.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.534+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:53.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.534+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4050970 con 0x7fcfe4044cf0 2024-04-27T04:14:53.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.534+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052a60 con 0x7fcff8101eb0 2024-04-27T04:14:53.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.534+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:14:53.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.534+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:14:53.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.535+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:53.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.535+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052ba0 con 0x7fcfe4044cf0 2024-04-27T04:14:53.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.535+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803bef0 tx=0x7fcfe802fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:14:53.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.535+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4044cf0 2024-04-27T04:14:53.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.536+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4044cf0 2024-04-27T04:14:53.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:53.537+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe804d620 con 0x7fcfe4044cf0 2024-04-27T04:14:53.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:14:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:14:53.534+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5191 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:14:54.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:54.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:54.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:54.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:54.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:54.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:54.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:54.064+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:54.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:53.953+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:54.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:53.953+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:54.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:53.953+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:54.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:53.954+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:54.058+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:54.058+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:54.058+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:54.059+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:55.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:55.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:55.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:55.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:55.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:55.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:55.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:55.106+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:54.927+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:54.927+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:54.927+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:54.928+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:55.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:55.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:55.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:55.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:55.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:55.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:55.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:55.077+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:56.403 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:56.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:56.404 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:56.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:56.404 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:56.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:56.404 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:56.081+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:56.404 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:55.975+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:56.404 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:55.975+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:56.404 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:55.975+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:56.404 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:55.976+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:56.405 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:56.119+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:56.405 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:56.119+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:56.405 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:56.119+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:56.405 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:56.120+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:57.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:56.965+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:57.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:56.965+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:57.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:56.965+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:57.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:56.966+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:57.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:57.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:57.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:57.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:57.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:57.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:57.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:57.110+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:57.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:57.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:57.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:57.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:57.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:57.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:57.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:57.142+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:58.394 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:57.934+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:58.395 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:57.934+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:58.395 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:57.934+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:58.395 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:57.935+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:58.395 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:58.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:58.395 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:58.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:58.395 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:58.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:58.395 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:58.067+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:58.396 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:58.149+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:58.396 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:58.149+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:58.396 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:58.149+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:58.396 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:58.150+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:58.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.534+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:14:58.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.535+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:14:58.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.535+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803bef0 tx=0x7fcfe802fba0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:14:58.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.535+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803bef0 tx=0x7fcfe802fba0 comp rx=0 tx=0).stop 2024-04-27T04:14:58.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.535+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:14:58.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.535+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:58.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.535+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:58.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.535+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:14:58.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.535+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4050970 con 0x7fcfe4054300 2024-04-27T04:14:58.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.535+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052a60 con 0x7fcff8101eb0 2024-04-27T04:14:58.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.535+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:14:58.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.536+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:14:58.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.536+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:14:58.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.536+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052ba0 con 0x7fcfe4054300 2024-04-27T04:14:58.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.537+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fcff003df60 tx=0x7fcfe803c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:14:58.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.537+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003ee80 con 0x7fcfe4054300 2024-04-27T04:14:58.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.537+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0009e80 con 0x7fcfe4054300 2024-04-27T04:14:58.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:58.539+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4054300 2024-04-27T04:14:58.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:14:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:14:58.535+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5255 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:14:59.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:58.963+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:59.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:58.963+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:59.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:58.963+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:14:59.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:58.963+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:14:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:59.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:59.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:59.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:14:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:14:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:14:59.060+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:59.152+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:59.152+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:59.152+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:14:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:14:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:14:59.153+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:14:59.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:59.477+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:14:59.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:14:59.477+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:15:00.192 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:59.928+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:00.192 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:59.928+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:00.192 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:59.928+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:00.192 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:14:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:14:59.929+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:00.192 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:00.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:00.192 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:00.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:00.192 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:00.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:00.192 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:00.029+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:00.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:00.193+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:00.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:00.193+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:00.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:00.193+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:00.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:00.194+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:01.150 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:00.894+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:01.150 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:00.894+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:01.150 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:00.894+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:01.150 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:00.895+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:01.150 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:01.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:01.151 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:01.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:01.151 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:01.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:01.151 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:01.030+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:01.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:01.151+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:01.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:01.151+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:01.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:01.151+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:01.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:01.152+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:02.180 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:01.923+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:02.181 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:01.923+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:02.181 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:01.923+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:02.181 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:01.924+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:02.181 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:02.038+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:02.181 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:02.038+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:02.181 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:02.038+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:02.181 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:02.039+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:02.182+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:02.182+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:02.182+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:02.183+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:03.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:03.133+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:03.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:03.133+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:03.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:03.133+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:03.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:03.134+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:03.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:03.003+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:03.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:03.003+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:03.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:03.003+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:03.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:03.004+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:03.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:02.894+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:03.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:02.894+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:03.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:02.894+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:03.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:02.895+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.535+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:15:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.535+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:15:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.535+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fcff003df60 tx=0x7fcfe803c550 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:15:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.536+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fcff003df60 tx=0x7fcfe803c550 comp rx=0 tx=0).stop 2024-04-27T04:15:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.536+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:15:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.536+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.536+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.536+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.536+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4050970 con 0x7fcfe4044cf0 2024-04-27T04:15:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.536+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052a60 con 0x7fcff8101eb0 2024-04-27T04:15:03.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.536+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:15:03.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.537+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:15:03.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.537+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:03.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.537+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052ba0 con 0x7fcfe4044cf0 2024-04-27T04:15:03.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.537+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fcff000d010 tx=0x7fcff004df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:15:03.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.538+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0025ba0 con 0x7fcfe4044cf0 2024-04-27T04:15:03.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.538+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0009e80 con 0x7fcfe4044cf0 2024-04-27T04:15:03.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:03.540+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4044cf0 2024-04-27T04:15:03.862 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:15:03.536+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5317 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:15:04.163 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:03.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:04.163 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:03.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:04.164 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:03.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:04.164 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:03.992+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:04.164 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:03.863+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:04.164 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:03.863+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:04.164 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:03.863+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:04.164 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:03.864+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:04.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:04.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:04.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:04.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:04.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:04.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:04.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:04.166+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:05.136 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:05.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:05.136 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:05.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:05.136 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:05.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:05.136 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:05.024+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:05.136 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:04.825+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:05.136 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:04.825+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:05.136 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:04.825+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:05.137 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:04.826+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:05.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:05.137+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:05.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:05.137+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:05.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:05.137+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:05.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:05.138+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:05.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:05.795+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:05.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:05.795+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:05.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:05.795+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:05.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:05.796+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:06.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:06.102+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:06.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:06.102+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:06.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:06.102+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:06.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:06.103+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:06.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:06.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:06.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:06.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:06.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:06.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:06.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:06.002+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:07.109 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:06.807+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:07.109 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:06.807+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:07.109 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:06.807+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:07.109 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:06.808+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:07.109 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:07.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:07.110 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:07.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:07.110 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:07.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:07.110 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:07.016+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:07.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:07.110+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:07.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:07.110+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:07.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:07.110+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:07.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:07.111+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:08.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:08.085+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:08.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:08.085+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:08.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:08.085+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:08.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:08.085+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:08.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:07.838+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:08.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:07.838+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:08.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:07.838+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:08.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:07.840+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:08.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:07.984+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:08.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:07.984+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:08.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:07.984+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:08.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:07.985+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:08.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.536+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:15:08.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.537+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:15:08.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.537+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fcff000d010 tx=0x7fcff004df30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:15:08.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.537+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fcff000d010 tx=0x7fcff004df30 comp rx=0 tx=0).stop 2024-04-27T04:15:08.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.537+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:15:08.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.537+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:08.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.537+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:08.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.537+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:08.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.537+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4050970 con 0x7fcfe4054300 2024-04-27T04:15:08.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.537+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052a60 con 0x7fcff8101eb0 2024-04-27T04:15:08.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.538+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:15:08.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.538+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:15:08.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.538+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:08.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.538+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052ba0 con 0x7fcfe4054300 2024-04-27T04:15:08.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.539+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803c350 tx=0x7fcff00068a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:15:08.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.541+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4054300 2024-04-27T04:15:08.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.541+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4054300 2024-04-27T04:15:08.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:08.542+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe804d620 con 0x7fcfe4054300 2024-04-27T04:15:08.860 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:15:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:15:08.537+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5380 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:15:09.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:09.018+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:09.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:09.018+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:09.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:09.018+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:09.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:09.019+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:09.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:08.861+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:09.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:08.861+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:09.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:08.861+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:09.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:08.863+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:09.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:09.043+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:09.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:09.043+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:09.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:09.043+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:09.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:09.044+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:09.814+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:09.814+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:09.814+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:09.815+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:10.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:10.027+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:10.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:10.027+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:10.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:10.027+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:10.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:10.027+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:10.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:10.053+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:10.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:10.053+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:10.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:10.053+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:10.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:10.054+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:11.025 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:11.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:11.025 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:11.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:11.025 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:11.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:11.025 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:11.026+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:11.025 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:11.025+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:11.025 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:11.025+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:11.025 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:11.025+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:11.026 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:10.776+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:11.026 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:10.776+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:11.026 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:10.776+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:11.026 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:10.776+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:11.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:11.026+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:11.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:11.774+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:11.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:11.774+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:11.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:11.774+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:11.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:11.775+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:12.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:12.015+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:12.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:12.015+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:12.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:12.015+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:12.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:12.016+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:12.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:12.072+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:12.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:12.072+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:12.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:12.072+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:12.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:12.073+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:13.035 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:12.756+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:13.036 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:12.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:13.036 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:12.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:13.036 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:12.757+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:13.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:13.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:13.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:13.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:13.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:13.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:13.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:13.101+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:13.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:13.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:13.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:13.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:13.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:13.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:13.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:13.037+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:13.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.537+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:15:13.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.537+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:15:13.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.537+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803c350 tx=0x7fcff00068a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:15:13.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.537+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fcfe803c350 tx=0x7fcff00068a0 comp rx=0 tx=0).stop 2024-04-27T04:15:13.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.537+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:15:13.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.537+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:13.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.537+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:13.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.538+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:13.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.538+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b30 con 0x7fcfe4044cf0 2024-04-27T04:15:13.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.538+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c70 con 0x7fcff8101eb0 2024-04-27T04:15:13.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.538+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:15:13.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.539+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:15:13.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.539+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:13.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.539+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052db0 con 0x7fcfe4044cf0 2024-04-27T04:15:13.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.539+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4007c10 tx=0x7fcfe8038d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:15:13.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.540+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0025ba0 con 0x7fcfe4044cf0 2024-04-27T04:15:13.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.541+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0005db0 con 0x7fcfe4044cf0 2024-04-27T04:15:13.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:13.542+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4044cf0 2024-04-27T04:15:13.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:15:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:15:13.538+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5443 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:15:13.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:13.782+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:13.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:13.782+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:13.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:13.782+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:13.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:13.783+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:14.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:14.119+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:14.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:14.119+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:14.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:14.119+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:14.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:14.120+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:14.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:14.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:14.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:14.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:14.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:14.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:14.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:14.077+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:14.837+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:15:14.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:14.837+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:15:15.084 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:14.798+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:15.085 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:14.798+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:15.085 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:14.798+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:15.085 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:14.799+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:15.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:15.161+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:15.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:15.161+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:15.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:15.161+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:15.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:15.162+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:15.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:15.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:15.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:15.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:15.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:15.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:15.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:15.087+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:16.074 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:15.818+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:16.074 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:15.818+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:16.074 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:15.818+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:16.075 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:15.819+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:16.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:16.075+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:16.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:16.075+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:16.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:16.075+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:16.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:16.076+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:16.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:16.169+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:16.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:16.169+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:16.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:16.169+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:16.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:16.170+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:17.102 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:16.785+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:17.102 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:16.785+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:17.102 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:16.785+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:17.103 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:16.786+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:17.103+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:17.103+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:17.103+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:17.104+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:17.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:17.152+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:17.152+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:17.152+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:17.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:17.152+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:18.115 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:17.821+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:18.115 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:17.821+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:18.116 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:17.821+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:18.116 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:17.822+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:18.116+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:18.117+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:18.117+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:18.117+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:18.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:18.199+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:18.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:18.199+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:18.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:18.199+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:18.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:18.200+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:18.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.538+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:15:18.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.538+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:15:18.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.538+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4007c10 tx=0x7fcfe8038d30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:15:18.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.538+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4007c10 tx=0x7fcfe8038d30 comp rx=0 tx=0).stop 2024-04-27T04:15:18.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.538+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:15:18.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.538+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:18.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.538+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:18.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.538+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:18.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.538+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b30 con 0x7fcfe4054300 2024-04-27T04:15:18.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.538+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c70 con 0x7fcff8101eb0 2024-04-27T04:15:18.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.539+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:15:18.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.539+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:15:18.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.539+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:18.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.539+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052db0 con 0x7fcfe4054300 2024-04-27T04:15:18.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.540+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400af20 tx=0x7fcff004e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:15:18.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.541+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4054300 2024-04-27T04:15:18.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.542+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4054300 2024-04-27T04:15:18.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:18.542+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe804d620 con 0x7fcfe4054300 2024-04-27T04:15:18.822 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:15:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:15:18.539+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5507 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:15:19.132 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:18.823+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:19.132 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:18.823+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:19.132 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:18.823+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:19.132 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:18.824+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:19.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:19.183+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:19.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:19.183+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:19.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:19.183+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:19.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:19.183+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:19.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:19.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:19.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:19.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:19.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:19.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:19.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:19.134+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:20.104 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:19.784+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:20.104 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:19.784+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:20.104 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:19.784+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:20.104 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:19.785+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:20.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:20.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:20.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:20.106+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:20.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:20.219+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:20.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:20.219+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:20.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:20.219+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:20.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:20.220+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:21.090 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:20.770+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:21.090 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:20.770+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:21.091 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:20.770+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:21.091 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:20.771+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:21.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:21.092+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:21.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:21.092+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:21.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:21.092+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:21.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:21.092+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:21.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:21.231+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:21.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:21.231+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:21.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:21.231+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:21.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:21.232+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:21.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:21.791+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:21.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:21.791+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:21.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:21.791+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:21.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:21.792+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:22.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:22.085+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:22.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:22.085+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:22.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:22.085+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:22.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:22.086+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:22.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:22.225+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:22.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:22.225+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:22.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:22.225+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:22.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:22.226+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:23.122 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:22.743+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:23.122 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:22.743+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:23.122 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:22.743+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:23.122 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:22.744+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:23.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:23.228+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:23.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:23.228+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:23.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:23.228+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:23.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:23.229+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:23.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:23.123+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:23.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:23.123+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:23.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:23.123+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:23.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:23.124+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:23.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.539+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:15:23.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.539+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:15:23.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.539+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400af20 tx=0x7fcff004e3c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:15:23.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.539+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400af20 tx=0x7fcff004e3c0 comp rx=0 tx=0).stop 2024-04-27T04:15:23.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.539+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:15:23.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.539+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe40447b0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:23.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.539+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:23.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.539+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:23.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.539+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b30 con 0x7fcfe4044cf0 2024-04-27T04:15:23.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.539+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c70 con 0x7fcff8101eb0 2024-04-27T04:15:23.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.540+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:15:23.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.540+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:15:23.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.540+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:23.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.540+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052db0 con 0x7fcfe4044cf0 2024-04-27T04:15:23.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.541+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8040040 tx=0x7fcfe8043010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:15:23.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.541+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe801bc30 con 0x7fcfe4044cf0 2024-04-27T04:15:23.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.541+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4044cf0 2024-04-27T04:15:23.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:23.542+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013c60 con 0x7fcfe4044cf0 2024-04-27T04:15:23.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:15:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:15:23.540+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5570 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:15:23.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:23.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:23.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:23.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:23.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:23.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:23.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:23.708+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:24.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:24.211+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:24.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:24.211+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:24.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:24.211+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:24.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:24.212+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:24.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:24.101+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:24.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:24.101+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:24.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:24.101+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:24.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:24.102+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:25.138 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:24.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:25.138 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:24.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:25.138 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:24.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:25.138 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:24.747+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:25.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:25.261+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:25.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:25.261+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:25.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:25.261+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:25.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:25.263+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:25.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:25.139+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:25.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:25.139+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:25.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:25.139+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:25.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:25.140+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:26.146 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:25.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:26.146 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:25.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:26.146 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:25.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:26.147 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:25.706+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:26.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:26.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:26.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:26.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:26.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:26.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:26.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:26.259+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:26.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:26.148+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:26.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:26.148+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:26.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:26.148+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:26.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:26.148+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:27.132 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:26.739+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:27.133 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:26.739+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:27.133 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:26.739+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:27.133 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:26.740+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:27.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:27.233+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:27.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:27.233+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:27.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:27.233+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:27.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:27.234+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:27.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:27.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:27.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:27.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:27.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:27.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:27.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:27.134+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:28.146 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:27.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:28.146 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:27.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:28.146 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:27.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:28.146 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:27.732+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:28.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:28.147+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:28.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:28.147+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:28.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:28.147+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:28.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:28.148+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:28.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:28.237+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:28.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:28.237+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:28.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:28.237+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:28.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:28.237+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:28.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.540+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:15:28.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.540+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:15:28.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.540+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8040040 tx=0x7fcfe8043010 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:15:28.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.540+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8040040 tx=0x7fcfe8043010 comp rx=0 tx=0).stop 2024-04-27T04:15:28.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.540+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:15:28.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.540+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4044cf0 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:28.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.540+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:28.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.540+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:28.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.540+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b30 con 0x7fcfe4054300 2024-04-27T04:15:28.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.540+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c70 con 0x7fcff8101eb0 2024-04-27T04:15:28.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.541+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4051f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:15:28.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.541+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe40447b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:15:28.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.541+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe40447b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:28.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.542+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052db0 con 0x7fcfe4054300 2024-04-27T04:15:28.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.542+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4051f10 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fcff000bfd0 tx=0x7fcfe800b300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:15:28.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.544+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0025ba0 con 0x7fcfe4054300 2024-04-27T04:15:28.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.544+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff003a520 con 0x7fcfe4054300 2024-04-27T04:15:28.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:28.544+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4054300 2024-04-27T04:15:28.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:15:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:15:28.541+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5634 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:15:28.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:28.781+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:28.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:28.781+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:28.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:28.781+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:28.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:28.782+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:29.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:29.234+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:29.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:29.234+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:29.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:29.234+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:29.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:29.235+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:29.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:29.113+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:29.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:29.113+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:29.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:29.113+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:29.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:29.114+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:30.138 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:29.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:30.138 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:29.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:30.138 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:29.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:30.139 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:29.761+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:30.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:30.197+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:15:30.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:30.197+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:15:30.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:30.255+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:30.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:30.255+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:30.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:30.255+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:30.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:30.256+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:30.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:30.139+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:30.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:30.139+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:30.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:30.139+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:30.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:30.140+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:31.145 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:30.799+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:31.145 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:30.799+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:31.145 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:30.799+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:31.145 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:30.800+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:31.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:31.229+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:31.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:31.229+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:31.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:31.229+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:31.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:31.230+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:31.432 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:31.144+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:31.432 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:31.144+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:31.432 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:31.144+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:31.432 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:31.144+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:31.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:31.794+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:31.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:31.794+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:31.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:31.794+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:31.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:31.796+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:32.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:32.183+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:32.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:32.183+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:32.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:32.183+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:32.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:32.184+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:32.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:32.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:32.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:32.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:32.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:32.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:32.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:32.184+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:33.146 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:32.784+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:33.147 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:32.784+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:33.147 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:32.784+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:33.147 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:32.785+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:33.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:33.148+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:33.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:33.148+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:33.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:33.148+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:33.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:33.149+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:33.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:33.196+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:33.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:33.196+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:33.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:33.196+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:33.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:33.197+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:33.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.541+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4051f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:15:33.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.541+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4051f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:15:33.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.541+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4051f10 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fcff000bfd0 tx=0x7fcfe800b300 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:15:33.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.541+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4051f10 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fcff000bfd0 tx=0x7fcfe800b300 comp rx=0 tx=0).stop 2024-04-27T04:15:33.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.541+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:15:33.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.541+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4051f10 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:33.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.541+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:33.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.541+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:33.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.541+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b30 con 0x7fcfe40447b0 2024-04-27T04:15:33.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.541+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c70 con 0x7fcff8101eb0 2024-04-27T04:15:33.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.541+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:15:33.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.542+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:15:33.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.542+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4052450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:33.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.542+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052db0 con 0x7fcfe40447b0 2024-04-27T04:15:33.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.543+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4050f60 tx=0x7fcff0013f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:15:33.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.544+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0013960 con 0x7fcfe40447b0 2024-04-27T04:15:33.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.544+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0013ac0 con 0x7fcfe40447b0 2024-04-27T04:15:33.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:33.545+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059040 con 0x7fcfe40447b0 2024-04-27T04:15:33.794 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:15:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:15:33.542+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5698 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:15:33.794 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:33.790+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:33.794 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:33.790+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:33.795 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:33.790+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:33.795 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:33.790+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:34.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:34.149+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:34.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:34.149+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:34.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:34.149+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:34.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:34.150+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:34.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:34.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:34.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:34.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:34.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:34.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:34.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:34.175+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:35.153 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:34.756+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:35.153 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:34.756+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:35.153 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:34.756+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:35.153 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:34.757+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:35.169+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:35.169+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:35.169+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:35.170+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:35.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:35.154+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:35.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:35.154+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:35.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:35.154+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:35.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:35.155+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:36.156 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:35.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:36.156 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:35.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:36.156 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:35.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:36.157 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:35.749+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:36.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:36.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:36.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:36.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:36.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:36.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:36.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:36.204+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:36.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:36.158+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:36.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:36.158+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:36.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:36.158+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:36.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:36.158+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:37.110 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:36.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:37.110 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:36.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:37.110 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:36.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:37.110 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:36.707+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:37.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:37.238+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:37.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:37.238+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:37.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:37.238+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:37.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:37.239+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:37.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:37.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:37.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:37.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:37.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:37.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:37.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:37.112+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:38.091 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:37.742+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:38.091 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:37.742+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:38.091 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:37.742+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:38.091 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:37.743+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:38.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:38.247+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:38.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:38.247+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:38.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:38.247+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:38.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:38.247+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:38.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:38.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:38.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:38.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:38.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:38.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:38.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:38.093+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:38.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.542+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:15:38.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.542+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:15:38.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.542+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4050f60 tx=0x7fcff0013f50 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:15:38.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.542+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4050f60 tx=0x7fcff0013f50 comp rx=0 tx=0).stop 2024-04-27T04:15:38.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.542+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:15:38.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.542+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:38.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.542+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:38.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.542+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:38.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.542+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b30 con 0x7fcfe4054300 2024-04-27T04:15:38.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.542+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c70 con 0x7fcff8101eb0 2024-04-27T04:15:38.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.543+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:15:38.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.543+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:15:38.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.543+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:38.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.543+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052db0 con 0x7fcfe4054300 2024-04-27T04:15:38.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.544+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4017ca0 tx=0x7fcff004dff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:15:38.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.545+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0013960 con 0x7fcfe4054300 2024-04-27T04:15:38.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.545+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0036a10 con 0x7fcfe4054300 2024-04-27T04:15:38.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:38.545+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059050 con 0x7fcfe4054300 2024-04-27T04:15:38.847 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:15:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:15:38.543+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5763 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:15:38.847 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:38.756+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:38.847 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:38.756+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:38.847 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:38.756+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:38.847 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:38.757+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:39.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:39.070+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:39.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:39.070+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:39.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:39.070+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:39.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:39.071+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:39.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:39.209+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:39.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:39.209+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:39.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:39.209+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:39.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:39.210+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:40.076 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:39.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:40.077 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:39.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:40.077 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:39.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:40.077 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:39.738+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:40.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:40.241+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:40.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:40.241+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:40.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:40.241+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:40.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:40.241+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:40.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:40.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:40.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:40.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:40.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:40.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:40.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:40.077+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:41.053 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:40.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:41.053 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:40.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:41.053 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:40.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:41.053 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:40.720+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:41.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:41.211+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:41.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:41.211+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:41.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:41.211+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:41.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:41.211+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:41.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:41.053+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:41.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:41.053+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:41.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:41.053+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:41.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:41.054+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:41.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:41.727+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:41.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:41.727+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:41.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:41.727+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:41.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:41.728+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:42.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:42.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:42.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:42.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:42.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:42.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:42.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:42.084+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:42.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:42.209+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:42.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:42.209+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:42.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:42.209+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:42.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:42.210+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:42.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:42.695+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:42.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:42.695+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:42.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:42.695+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:42.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:42.696+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:43.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:43.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:43.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:43.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:43.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:43.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:43.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:43.068+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:43.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:43.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:43.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:43.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:43.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:43.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:43.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:43.222+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:43.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.542+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:15:43.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.542+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:15:43.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.542+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4017ca0 tx=0x7fcff004dff0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:15:43.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.542+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4017ca0 tx=0x7fcff004dff0 comp rx=0 tx=0).stop 2024-04-27T04:15:43.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.542+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:15:43.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.542+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:43.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.542+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:43.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.542+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:43.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.542+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcfe40447b0 2024-04-27T04:15:43.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.542+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b30 con 0x7fcff8101eb0 2024-04-27T04:15:43.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.543+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:15:43.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.543+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:15:43.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.543+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:43.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.543+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052c70 con 0x7fcfe40447b0 2024-04-27T04:15:43.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.544+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4050af0 tx=0x7fcff005d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:15:43.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.544+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe40447b0 2024-04-27T04:15:43.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.545+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe40447b0 2024-04-27T04:15:43.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:43.546+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059040 con 0x7fcfe40447b0 2024-04-27T04:15:43.900 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:15:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:15:43.543+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5825 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:15:43.901 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:43.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:43.901 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:43.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:43.901 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:43.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:43.901 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:43.692+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:44.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:44.116+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:44.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:44.116+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:44.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:44.116+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:44.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:44.117+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:44.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:44.271+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:44.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:44.271+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:44.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:44.271+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:44.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:44.272+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:45.097 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:44.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:45.097 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:44.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:45.097 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:44.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:45.097 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:44.731+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:45.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:45.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:45.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:45.311+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:45.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:45.097+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:45.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:45.097+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:45.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:45.097+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:45.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:45.098+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:45.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:45.556+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:15:45.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:45.556+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:15:45.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:45.747+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:45.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:45.747+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:45.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:45.747+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:45.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:45.748+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:46.338 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:46.337+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:46.338 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:46.337+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:46.338 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:46.337+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:46.338 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:46.088+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:46.339 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:46.088+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:46.339 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:46.088+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:46.339 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:46.088+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:46.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:46.338+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:47.089 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:46.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:47.090 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:46.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:47.090 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:46.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:47.090 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:46.701+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:47.307+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:47.307+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:47.307+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:47.308+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:47.089+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:47.089+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:47.089+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:47.090+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:47.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:47.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:47.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:47.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:47.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:47.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:47.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:47.669+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:48.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:48.088+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:48.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:48.088+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:48.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:48.088+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:48.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:48.089+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.543+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:15:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.543+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:15:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.543+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4050af0 tx=0x7fcff005d040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:15:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.543+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4050af0 tx=0x7fcff005d040 comp rx=0 tx=0).stop 2024-04-27T04:15:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.543+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:15:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.543+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.543+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.543+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.543+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b30 con 0x7fcfe4054300 2024-04-27T04:15:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.543+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c70 con 0x7fcff8101eb0 2024-04-27T04:15:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.543+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:15:48.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.544+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:15:48.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.544+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:48.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.544+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052db0 con 0x7fcfe4054300 2024-04-27T04:15:48.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.545+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400a230 tx=0x7fcff003ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:15:48.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.545+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe801bc30 con 0x7fcfe4054300 2024-04-27T04:15:48.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.546+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4054300 2024-04-27T04:15:48.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:48.547+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe804ba40 con 0x7fcfe4054300 2024-04-27T04:15:48.628 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:15:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:15:48.543+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5888 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:15:48.628 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:48.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:48.629 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:48.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:48.629 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:48.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:48.629 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:48.341+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:48.959 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:48.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:48.959 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:48.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:48.959 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:48.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:48.959 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:48.707+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:49.367 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:49.106+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:49.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:49.106+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:49.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:49.106+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:49.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:49.107+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:49.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:49.368+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:49.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:49.368+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:49.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:49.368+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:49.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:49.368+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:50.134 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:49.705+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:50.134 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:49.705+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:50.134 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:49.705+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:50.134 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:49.707+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:50.400 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:50.134+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:50.400 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:50.134+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:50.400 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:50.134+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:50.400 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:50.135+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:50.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:50.400+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:50.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:50.400+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:50.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:50.400+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:50.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:50.401+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:51.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:50.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:51.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:50.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:51.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:50.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:51.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:50.704+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:51.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:51.400+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:51.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:51.400+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:51.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:51.400+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:51.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:51.401+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:51.569 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:51.174+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:51.569 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:51.174+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:51.569 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:51.174+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:51.569 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:51.175+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:51.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:51.730+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:51.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:51.730+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:51.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:51.730+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:51.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:51.730+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:52.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:52.200+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:52.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:52.200+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:52.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:52.200+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:52.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:52.201+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:52.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:52.381+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:52.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:52.381+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:52.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:52.381+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:52.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:52.381+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:53.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:52.695+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:53.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:52.695+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:53.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:52.695+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:53.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:52.695+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.545+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:15:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.545+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:15:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.545+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400a230 tx=0x7fcff003ac60 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:15:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.545+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400a230 tx=0x7fcff003ac60 comp rx=0 tx=0).stop 2024-04-27T04:15:53.544 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:53.247+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:53.544 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:53.247+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:53.544 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:53.247+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:53.544 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:53.249+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:53.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.545+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:15:53.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.545+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:53.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.545+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:53.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.545+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:53.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.545+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b30 con 0x7fcfe40447b0 2024-04-27T04:15:53.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.545+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c70 con 0x7fcff8101eb0 2024-04-27T04:15:53.546 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:53.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:53.546 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:53.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:53.546 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:53.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:53.546 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:53.390+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:53.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.545+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:15:53.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.546+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:15:53.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.546+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:53.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.546+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052db0 con 0x7fcfe40447b0 2024-04-27T04:15:53.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.547+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40109b0 tx=0x7fcfe8014900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:15:53.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.547+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe801bc30 con 0x7fcfe40447b0 2024-04-27T04:15:53.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.547+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe40447b0 2024-04-27T04:15:53.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:53.549+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013510 con 0x7fcfe40447b0 2024-04-27T04:15:53.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:15:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:15:53.546+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 5950 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:15:53.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:53.651+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:53.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:53.651+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:53.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:53.651+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:53.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:53.651+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:54.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:54.251+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:54.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:54.251+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:54.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:54.251+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:54.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:54.252+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:54.771 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:54.675+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:54.772 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:54.675+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:54.772 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:54.675+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:54.772 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:54.677+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:54.772 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:54.438+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:54.772 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:54.438+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:54.773 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:54.438+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:54.773 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:54.439+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:55.630 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:55.433+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:55.630 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:55.433+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:55.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:55.433+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:55.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:55.434+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:55.631 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:55.246+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:55.631 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:55.246+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:55.631 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:55.246+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:55.631 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:55.247+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:55.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:55.632+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:55.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:55.632+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:55.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:55.632+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:55.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:55.632+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:56.628 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:56.210+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:56.629 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:56.210+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:56.629 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:56.210+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:56.629 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:56.211+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:56.630 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:56.417+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:56.630 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:56.417+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:56.630 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:56.417+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:56.630 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:56.418+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:56.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:56.629+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:56.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:56.629+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:56.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:56.629+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:56.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:56.630+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:57.637 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:57.404+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:57.637 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:57.404+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:57.638 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:57.404+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:57.638 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:57.405+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:57.638 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:57.207+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:57.639 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:57.207+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:57.639 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:57.207+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:57.639 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:57.208+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:57.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:57.638+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:57.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:57.638+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:57.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:57.638+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:57.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:57.639+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:58.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.546+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:15:58.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.546+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:15:58.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.546+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40109b0 tx=0x7fcfe8014900 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:15:58.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.546+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40109b0 tx=0x7fcfe8014900 comp rx=0 tx=0).stop 2024-04-27T04:15:58.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.546+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:15:58.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.546+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:58.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.546+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:58.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.546+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:15:58.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.546+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b10 con 0x7fcfe4054300 2024-04-27T04:15:58.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.546+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c50 con 0x7fcff8101eb0 2024-04-27T04:15:58.546 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:58.255+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:58.546 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:58.255+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:58.546 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:58.255+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:58.546 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:58.256+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:58.547 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:58.444+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:58.547 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:58.444+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:58.547 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:58.444+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:58.547 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:58.446+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:58.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.547+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:15:58.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.547+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:15:58.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.547+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:15:58.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.547+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052d90 con 0x7fcfe4054300 2024-04-27T04:15:58.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.547+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8008000 tx=0x7fcfe800b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:15:58.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.549+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe801bc30 con 0x7fcfe4054300 2024-04-27T04:15:58.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.549+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe803a810 con 0x7fcfe4054300 2024-04-27T04:15:58.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:15:58.550+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013580 con 0x7fcfe4054300 2024-04-27T04:15:58.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:15:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:15:58.548+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6015 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:15:58.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:58.657+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:58.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:58.657+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:58.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:58.657+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:58.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:58.658+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:59.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:59.302+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:59.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:59.302+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:59.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:59.302+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:15:59.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:15:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:15:59.303+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:15:59.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:59.685+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:59.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:59.685+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:59.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:59.685+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:15:59.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:15:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:15:59.686+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:15:59.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:59.446+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:59.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:59.446+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:59.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:59.446+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:15:59.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:15:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:15:59.447+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:00.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:00.411+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:00.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:00.411+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:00.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:00.411+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:00.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:00.412+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:00.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:00.352+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:00.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:00.352+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:00.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:00.352+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:00.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:00.353+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:00.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:00.917+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:16:00.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:00.917+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:16:01.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:00.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:01.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:00.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:01.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:00.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:01.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:00.701+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:01.442 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:01.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:01.443 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:01.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:01.443 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:01.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:01.443 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:01.347+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:01.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:01.444+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:01.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:01.444+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:01.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:01.444+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:01.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:01.444+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:01.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:01.656+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:01.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:01.656+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:01.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:01.656+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:01.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:01.657+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:02.335+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:02.335+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:02.335+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:02.336+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:02.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:02.478+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:02.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:02.478+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:02.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:02.478+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:02.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:02.480+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:03.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:02.698+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:03.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:02.698+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:03.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:02.698+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:03.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:02.699+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.548+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:16:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.548+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:16:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.548+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8008000 tx=0x7fcfe800b940 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:16:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.548+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8008000 tx=0x7fcfe800b940 comp rx=0 tx=0).stop 2024-04-27T04:16:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.548+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:16:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.548+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.548+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.548+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.548+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b10 con 0x7fcfe40447b0 2024-04-27T04:16:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.548+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c50 con 0x7fcff8101eb0 2024-04-27T04:16:03.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.548+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:16:03.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.549+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:16:03.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.549+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:03.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.549+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052d90 con 0x7fcfe40447b0 2024-04-27T04:16:03.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.549+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8030ea0 tx=0x7fcfe803a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:16:03.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.550+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe801bc30 con 0x7fcfe40447b0 2024-04-27T04:16:03.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.550+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804d7d0 con 0x7fcfe40447b0 2024-04-27T04:16:03.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:03.551+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013580 con 0x7fcfe40447b0 2024-04-27T04:16:03.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:16:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:16:03.549+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6077 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:16:03.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:03.360+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:03.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:03.360+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:03.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:03.360+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:03.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:03.362+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:03.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:03.442+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:03.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:03.442+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:03.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:03.442+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:03.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:03.443+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:03.987 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:03.733+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:03.987 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:03.733+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:03.988 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:03.733+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:03.988 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:03.733+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:04.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:04.365+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:04.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:04.365+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:04.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:04.365+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:04.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:04.367+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:04.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:04.436+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:04.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:04.436+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:04.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:04.436+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:04.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:04.436+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:05.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:04.694+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:05.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:04.694+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:05.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:04.694+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:05.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:04.694+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:05.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:05.362+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:05.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:05.362+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:05.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:05.362+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:05.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:05.364+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:05.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:05.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:05.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:05.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:05.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:05.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:05.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:05.389+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:05.681+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:05.681+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:05.681+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:05.682+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:06.664 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:06.401+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:06.664 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:06.401+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:06.664 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:06.401+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:06.664 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:06.402+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:06.664 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:06.392+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:06.664 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:06.392+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:06.664 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:06.392+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:06.664 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:06.392+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:06.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:06.665+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:06.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:06.665+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:06.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:06.665+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:06.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:06.666+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:07.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:07.381+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:07.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:07.381+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:07.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:07.381+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:07.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:07.381+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:07.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:07.356+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:07.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:07.356+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:07.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:07.356+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:07.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:07.357+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:08.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:07.698+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:08.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:07.698+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:08.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:07.698+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:08.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:07.699+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:08.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.549+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:16:08.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.549+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:16:08.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.549+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8030ea0 tx=0x7fcfe803a040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:16:08.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.549+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8030ea0 tx=0x7fcfe803a040 comp rx=0 tx=0).stop 2024-04-27T04:16:08.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.549+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:16:08.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.549+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:08.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.549+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:08.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.549+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:08.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.549+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052a60 con 0x7fcfe4054300 2024-04-27T04:16:08.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.549+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ba0 con 0x7fcff8101eb0 2024-04-27T04:16:08.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.550+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:16:08.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.551+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:16:08.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.551+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:08.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.551+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052ce0 con 0x7fcfe4054300 2024-04-27T04:16:08.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.551+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800c010 tx=0x7fcfe8044ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:16:08.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.552+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8038a00 con 0x7fcfe4054300 2024-04-27T04:16:08.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.553+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8038a00 con 0x7fcfe4054300 2024-04-27T04:16:08.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:08.554+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8044d10 con 0x7fcfe4054300 2024-04-27T04:16:08.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:16:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:16:08.550+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6140 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:16:08.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:08.353+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:08.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:08.353+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:08.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:08.353+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:08.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:08.355+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:08.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:08.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:08.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:08.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:08.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:08.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:08.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:08.378+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:09.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:08.727+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:09.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:08.727+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:09.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:08.727+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:09.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:08.727+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:09.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:09.370+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:09.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:09.370+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:09.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:09.370+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:09.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:09.371+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:09.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:09.334+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:09.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:09.334+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:09.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:09.334+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:09.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:09.334+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:09.768+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:09.768+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:09.768+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:09.769+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:10.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:10.348+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:10.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:10.348+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:10.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:10.348+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:10.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:10.349+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:10.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:10.355+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:10.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:10.355+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:10.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:10.355+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:10.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:10.356+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:11.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:10.761+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:11.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:10.761+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:11.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:10.761+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:11.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:10.762+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:11.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:11.351+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:11.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:11.351+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:11.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:11.351+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:11.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:11.352+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:11.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:11.327+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:11.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:11.327+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:11.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:11.327+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:11.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:11.328+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:12.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:11.789+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:12.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:11.789+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:12.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:11.789+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:12.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:11.791+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:12.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:12.388+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:12.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:12.388+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:12.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:12.388+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:12.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:12.389+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:12.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:12.342+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:12.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:12.342+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:12.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:12.342+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:12.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:12.343+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:13.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:12.809+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:13.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:12.809+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:13.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:12.809+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:13.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:12.811+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:13.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.550+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:16:13.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.550+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:16:13.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.550+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800c010 tx=0x7fcfe8044ca0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:16:13.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.550+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800c010 tx=0x7fcfe8044ca0 comp rx=0 tx=0).stop 2024-04-27T04:16:13.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.550+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:16:13.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.550+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:13.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.550+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:13.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.550+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:13.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.550+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052b30 con 0x7fcfe40447b0 2024-04-27T04:16:13.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.550+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c70 con 0x7fcff8101eb0 2024-04-27T04:16:13.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.551+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:16:13.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.551+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:16:13.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.551+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:13.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.552+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052db0 con 0x7fcfe40447b0 2024-04-27T04:16:13.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.552+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804f4d0 tx=0x7fcfe8017310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:16:13.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.553+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0010cd0 con 0x7fcfe40447b0 2024-04-27T04:16:13.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.553+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff00197a0 con 0x7fcfe40447b0 2024-04-27T04:16:13.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:13.554+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059040 con 0x7fcfe40447b0 2024-04-27T04:16:13.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:16:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:16:13.550+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6202 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:16:13.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:13.419+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:13.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:13.419+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:13.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:13.419+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:13.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:13.419+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:13.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:13.353+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:13.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:13.353+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:13.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:13.353+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:13.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:13.353+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:14.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:13.781+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:14.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:13.781+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:14.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:13.781+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:14.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:13.782+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:14.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:14.310+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:14.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:14.310+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:14.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:14.310+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:14.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:14.312+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:14.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:14.393+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:14.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:14.393+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:14.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:14.393+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:14.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:14.394+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:15.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:14.764+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:15.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:14.764+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:15.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:14.764+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:15.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:14.765+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:15.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:15.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:15.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:15.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:15.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:15.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:15.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:15.306+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:15.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:15.413+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:15.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:15.413+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:15.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:15.413+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:15.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:15.414+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:16.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:15.788+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:16.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:15.788+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:16.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:15.788+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:16.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:15.789+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:16.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:16.277+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:16:16.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:16.277+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:16:16.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:16.294+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:16.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:16.294+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:16.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:16.294+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:16.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:16.295+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:16.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:16.391+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:16.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:16.391+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:16.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:16.391+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:16.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:16.392+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:17.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:16.815+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:17.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:16.815+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:17.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:16.815+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:17.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:16.816+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:17.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:17.383+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:17.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:17.383+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:17.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:17.383+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:17.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:17.383+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:17.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:17.290+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:17.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:17.290+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:17.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:17.290+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:17.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:17.291+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:18.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:17.771+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:18.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:17.771+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:18.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:17.771+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:18.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:17.772+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:18.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.551+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:16:18.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.551+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:16:18.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.551+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804f4d0 tx=0x7fcfe8017310 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:16:18.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.551+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804f4d0 tx=0x7fcfe8017310 comp rx=0 tx=0).stop 2024-04-27T04:16:18.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.551+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:16:18.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.551+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:18.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.551+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:18.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.551+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:18.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.551+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052bc0 con 0x7fcfe4054300 2024-04-27T04:16:18.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.551+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052d00 con 0x7fcff8101eb0 2024-04-27T04:16:18.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.552+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:16:18.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.552+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:16:18.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.552+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:18.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.552+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e40 con 0x7fcfe4054300 2024-04-27T04:16:18.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.553+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fcff00013a0 tx=0x7fcff0042320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:16:18.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.555+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0010cd0 con 0x7fcfe4054300 2024-04-27T04:16:18.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.555+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0036840 con 0x7fcfe4054300 2024-04-27T04:16:18.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:18.556+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059040 con 0x7fcfe4054300 2024-04-27T04:16:18.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:16:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:16:18.551+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6266 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:16:18.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:18.326+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:18.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:18.326+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:18.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:18.326+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:18.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:18.327+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:18.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:18.338+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:18.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:18.338+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:18.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:18.338+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:18.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:18.339+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:19.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:18.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:19.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:18.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:19.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:18.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:19.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:18.749+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:19.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:19.333+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:19.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:19.333+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:19.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:19.333+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:19.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:19.334+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:19.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:19.364+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:19.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:19.364+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:19.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:19.364+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:19.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:19.365+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:20.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:19.736+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:20.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:19.736+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:20.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:19.736+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:20.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:19.738+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:20.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:20.376+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:20.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:20.376+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:20.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:20.376+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:20.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:20.377+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:20.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:20.353+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:20.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:20.353+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:20.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:20.353+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:20.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:20.354+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:21.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:20.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:21.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:20.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:21.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:20.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:21.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:20.693+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:21.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:21.398+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:21.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:21.398+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:21.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:21.398+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:21.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:21.399+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:21.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:21.369+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:21.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:21.369+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:21.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:21.369+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:21.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:21.370+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:22.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:21.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:22.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:21.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:22.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:21.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:22.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:21.701+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:22.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:22.397+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:22.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:22.397+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:22.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:22.397+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:22.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:22.398+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:22.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:22.388+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:22.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:22.388+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:22.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:22.388+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:22.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:22.388+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:23.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:22.726+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:23.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:22.726+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:23.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:22.726+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:23.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:22.728+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:23.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.552+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:16:23.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.552+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:16:23.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.552+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fcff00013a0 tx=0x7fcff0042320 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:16:23.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.552+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fcff00013a0 tx=0x7fcff0042320 comp rx=0 tx=0).stop 2024-04-27T04:16:23.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.552+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:16:23.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.552+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:23.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.552+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:23.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.552+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:23.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.552+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c70 con 0x7fcfe40447b0 2024-04-27T04:16:23.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.552+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052db0 con 0x7fcff8101eb0 2024-04-27T04:16:23.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.552+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:16:23.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.553+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:16:23.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.553+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:23.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.553+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052ef0 con 0x7fcfe40447b0 2024-04-27T04:16:23.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.554+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400f980 tx=0x7fcff002b5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:16:23.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.554+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe40447b0 2024-04-27T04:16:23.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.556+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe801bcd0 con 0x7fcfe40447b0 2024-04-27T04:16:23.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:23.556+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8044d10 con 0x7fcfe40447b0 2024-04-27T04:16:23.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:16:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:16:23.552+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6325 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:16:23.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:23.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:23.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:23.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:23.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:23.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:23.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:23.408+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:23.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:23.420+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:23.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:23.420+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:23.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:23.420+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:23.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:23.421+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:24.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:23.688+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:24.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:23.688+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:24.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:23.688+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:24.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:23.688+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:24.650 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:24.455+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:24.650 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:24.455+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:24.650 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:24.455+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:24.650 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:24.456+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:24.651 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:24.441+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:24.651 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:24.441+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:24.651 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:24.441+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:24.651 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:24.442+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:24.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:24.651+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:24.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:24.651+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:24.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:24.651+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:24.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:24.652+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:25.452+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:25.452+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:25.452+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:25.453+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:25.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:25.607+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:25.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:25.607+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:25.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:25.607+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:25.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:25.608+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:25.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:25.491+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:25.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:25.491+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:25.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:25.491+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:25.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:25.492+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:26.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:26.430+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:26.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:26.430+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:26.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:26.430+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:26.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:26.431+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:26.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:26.605+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:26.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:26.605+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:26.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:26.605+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:26.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:26.606+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:26.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:26.452+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:26.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:26.452+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:26.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:26.452+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:26.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:26.453+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:27.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:27.608+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:27.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:27.608+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:27.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:27.608+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:27.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:27.609+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:27.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:27.468+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:27.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:27.468+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:27.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:27.468+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:27.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:27.469+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:27.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:27.479+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:27.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:27.479+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:27.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:27.479+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:27.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:27.480+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:28.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.553+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:16:28.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.553+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:16:28.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.553+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400f980 tx=0x7fcff002b5f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:16:28.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.553+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400f980 tx=0x7fcff002b5f0 comp rx=0 tx=0).stop 2024-04-27T04:16:28.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.553+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:16:28.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.553+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:28.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.553+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:28.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.553+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:28.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.553+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c70 con 0x7fcfe4054300 2024-04-27T04:16:28.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.553+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052db0 con 0x7fcff8101eb0 2024-04-27T04:16:28.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.553+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:16:28.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.554+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:16:28.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.554+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:28.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.554+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052ef0 con 0x7fcfe4054300 2024-04-27T04:16:28.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.554+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fcff001c520 tx=0x7fcfe8026960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:16:28.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.554+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0010cd0 con 0x7fcfe4054300 2024-04-27T04:16:28.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.554+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe4054300 2024-04-27T04:16:28.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:28.556+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059040 con 0x7fcfe4054300 2024-04-27T04:16:28.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:16:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:16:28.553+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6378 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:16:28.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:28.510+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:28.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:28.510+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:28.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:28.510+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:28.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:28.510+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:28.467+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:28.467+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:28.467+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:28.468+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:28.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:28.634+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:28.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:28.634+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:28.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:28.634+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:28.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:28.635+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:29.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:29.488+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:29.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:29.488+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:29.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:29.488+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:29.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:29.489+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:29.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:29.533+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:29.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:29.533+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:29.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:29.533+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:29.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:29.534+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:29.649+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:29.649+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:29.649+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:29.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:29.650+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:30.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:30.638+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:30.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:30.638+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:30.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:30.638+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:30.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:30.639+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:30.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:30.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:30.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:30.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:30.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:30.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:30.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:30.562+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:30.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:30.498+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:30.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:30.498+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:30.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:30.498+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:30.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:30.498+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:31.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:31.637+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:16:31.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:31.637+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:16:31.822 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:31.456+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:31.822 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:31.456+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:31.823 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:31.456+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:31.823 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:31.457+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:31.823 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:31.629+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:31.823 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:31.629+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:31.823 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:31.629+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:31.823 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:31.630+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:31.823 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:31.536+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:31.824 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:31.536+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:31.824 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:31.536+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:31.824 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:31.537+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:32.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:32.494+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:32.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:32.494+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:32.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:32.494+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:32.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:32.495+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:32.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:32.597+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:32.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:32.597+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:32.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:32.597+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:32.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:32.598+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:32.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:32.520+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:32.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:32.520+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:32.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:32.520+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:32.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:32.521+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:33.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.554+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:16:33.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.554+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:16:33.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.554+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fcff001c520 tx=0x7fcfe8026960 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:16:33.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.554+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fcff001c520 tx=0x7fcfe8026960 comp rx=0 tx=0).stop 2024-04-27T04:16:33.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.554+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:16:33.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.554+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:33.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.554+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:33.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.554+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:33.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.554+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052be0 con 0x7fcfe40447b0 2024-04-27T04:16:33.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.554+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052d20 con 0x7fcff8101eb0 2024-04-27T04:16:33.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.554+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:16:33.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.555+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:16:33.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.555+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:33.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.555+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e60 con 0x7fcfe40447b0 2024-04-27T04:16:33.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.555+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4043e00 tx=0x7fcff000e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:16:33.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.556+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe40447b0 2024-04-27T04:16:33.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.556+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8039000 con 0x7fcfe40447b0 2024-04-27T04:16:33.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:33.557+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe804d9d0 con 0x7fcfe40447b0 2024-04-27T04:16:33.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:16:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:16:33.554+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6431 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:16:33.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:33.496+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:33.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:33.496+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:33.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:33.496+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:33.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:33.497+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:33.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:33.560+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:33.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:33.560+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:33.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:33.560+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:33.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:33.561+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:33.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:33.545+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:33.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:33.545+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:33.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:33.545+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:33.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:33.545+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:34.707 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:34.529+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:34.708 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:34.529+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:34.708 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:34.529+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:34.708 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:34.530+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:34.708 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:34.520+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:34.708 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:34.520+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:34.708 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:34.520+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:34.708 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:34.521+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:34.709 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:34.595+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:34.709 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:34.595+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:34.709 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:34.595+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:34.709 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:34.595+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:35.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:35.570+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:35.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:35.570+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:35.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:35.570+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:35.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:35.571+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:35.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:35.586+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:35.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:35.586+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:35.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:35.586+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:35.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:35.587+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:35.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:35.485+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:35.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:35.485+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:35.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:35.485+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:35.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:35.486+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:36.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:36.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:36.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:36.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:36.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:36.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:36.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:36.466+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:36.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:36.597+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:36.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:36.597+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:36.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:36.597+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:36.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:36.597+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:36.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:36.551+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:36.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:36.551+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:36.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:36.551+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:36.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:36.552+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:37.714 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:37.606+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:37.714 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:37.606+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:37.714 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:37.606+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:37.714 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:37.607+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:37.714 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:37.444+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:37.714 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:37.444+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:37.714 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:37.444+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:37.714 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:37.445+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:37.715 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:37.523+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:37.715 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:37.523+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:37.715 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:37.523+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:37.715 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:37.524+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.555+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:16:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.555+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:16:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.555+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4043e00 tx=0x7fcff000e420 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:16:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.555+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4043e00 tx=0x7fcff000e420 comp rx=0 tx=0).stop 2024-04-27T04:16:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.555+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:16:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.555+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.555+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.555+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.555+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052c00 con 0x7fcfe4054300 2024-04-27T04:16:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.555+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052d40 con 0x7fcff8101eb0 2024-04-27T04:16:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.555+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:16:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.555+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:16:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.555+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.556+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052e80 con 0x7fcfe4054300 2024-04-27T04:16:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.556+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80391e0 tx=0x7fcfe8046b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:16:38.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.557+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4054300 2024-04-27T04:16:38.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.558+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe80465e0 con 0x7fcfe4054300 2024-04-27T04:16:38.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:38.558+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe804d9d0 con 0x7fcfe4054300 2024-04-27T04:16:38.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:16:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:16:38.555+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6485 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:16:38.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:38.568+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:38.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:38.568+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:38.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:38.568+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:38.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:38.569+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:38.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:38.479+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:38.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:38.479+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:38.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:38.479+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:38.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:38.480+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:38.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:38.622+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:38.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:38.622+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:38.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:38.622+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:38.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:38.622+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:39.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:39.490+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:39.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:39.490+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:39.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:39.490+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:39.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:39.491+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:39.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:39.588+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:39.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:39.588+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:39.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:39.588+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:39.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:39.589+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:39.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:39.589+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:39.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:39.589+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:39.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:39.589+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:39.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:39.589+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:40.852 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:40.628+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:40.852 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:40.628+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:40.852 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:40.628+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:40.852 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:40.628+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:40.853 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:40.492+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:40.853 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:40.492+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:40.853 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:40.492+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:40.853 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:40.493+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:40.853 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:40.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:40.853 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:40.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:40.853 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:40.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:40.854 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:40.543+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:41.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:41.495+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:41.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:41.495+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:41.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:41.495+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:41.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:41.496+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:41.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:41.666+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:41.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:41.666+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:41.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:41.666+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:41.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:41.667+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:41.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:41.531+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:41.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:41.531+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:41.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:41.531+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:41.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:41.531+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:42.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:42.516+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:42.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:42.516+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:42.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:42.516+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:42.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:42.516+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:42.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:42.512+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:42.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:42.512+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:42.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:42.512+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:42.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:42.513+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:42.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:42.672+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:42.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:42.672+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:42.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:42.672+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:42.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:42.673+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.555+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:16:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.556+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:16:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.556+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80391e0 tx=0x7fcfe8046b70 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:16:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.556+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80391e0 tx=0x7fcfe8046b70 comp rx=0 tx=0).stop 2024-04-27T04:16:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.556+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:16:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.556+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.556+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.556+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.556+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052d60 con 0x7fcfe40447b0 2024-04-27T04:16:43.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.556+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ea0 con 0x7fcff8101eb0 2024-04-27T04:16:43.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.556+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:16:43.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.557+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:16:43.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.557+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:43.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.557+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052fe0 con 0x7fcfe40447b0 2024-04-27T04:16:43.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.558+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fcff002a210 tx=0x7fcfe8054860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:16:43.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.559+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0010cd0 con 0x7fcfe40447b0 2024-04-27T04:16:43.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.559+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe40447b0 2024-04-27T04:16:43.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:43.561+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe40447b0 2024-04-27T04:16:43.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:16:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:16:43.556+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6537 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:16:43.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:43.719+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:43.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:43.719+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:43.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:43.719+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:43.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:43.719+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:43.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:43.545+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:43.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:43.545+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:43.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:43.545+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:43.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:43.545+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:43.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:43.532+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:43.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:43.532+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:43.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:43.532+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:43.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:43.533+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:44.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:44.720+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:44.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:44.720+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:44.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:44.720+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:44.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:44.721+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:44.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:44.595+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:44.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:44.595+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:44.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:44.595+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:44.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:44.596+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:44.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:44.549+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:44.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:44.549+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:44.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:44.549+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:44.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:44.550+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:45.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:45.690+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:45.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:45.690+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:45.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:45.690+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:45.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:45.690+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:45.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:45.505+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:45.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:45.505+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:45.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:45.505+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:45.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:45.506+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:45.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:45.563+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:45.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:45.563+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:45.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:45.563+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:45.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:45.564+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:46.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:46.700+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:46.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:46.700+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:46.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:46.700+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:46.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:46.700+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:46.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:46.518+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:46.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:46.518+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:46.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:46.518+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:46.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:46.519+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:46.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:46.526+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:46.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:46.526+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:46.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:46.526+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:46.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:46.526+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:46.997+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:16:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:46.997+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:16:47.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:47.737+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:47.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:47.737+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:47.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:47.737+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:47.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:47.737+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:47.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:47.499+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:47.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:47.499+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:47.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:47.499+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:47.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:47.499+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:47.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:47.564+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:47.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:47.564+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:47.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:47.564+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:47.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:47.565+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:48.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.557+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:16:48.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.557+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:16:48.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.557+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fcff002a210 tx=0x7fcfe8054860 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:16:48.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.557+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fcff002a210 tx=0x7fcfe8054860 comp rx=0 tx=0).stop 2024-04-27T04:16:48.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.557+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:16:48.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.557+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:48.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.557+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:48.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.557+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:48.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.557+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ea0 con 0x7fcfe4054300 2024-04-27T04:16:48.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.557+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052fe0 con 0x7fcff8101eb0 2024-04-27T04:16:48.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.557+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:16:48.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.558+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:16:48.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.558+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:48.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.558+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4053120 con 0x7fcfe4054300 2024-04-27T04:16:48.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.558+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400f2c0 tx=0x7fcff0013980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:16:48.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.559+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0010cd0 con 0x7fcfe4054300 2024-04-27T04:16:48.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.561+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe4054300 2024-04-27T04:16:48.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:48.562+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4054300 2024-04-27T04:16:48.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:16:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:16:48.558+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6597 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:16:48.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:48.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:48.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:48.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:48.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:48.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:48.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:48.502+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:48.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:48.707+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:48.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:48.707+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:48.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:48.707+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:48.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:48.707+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:48.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:48.605+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:48.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:48.605+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:48.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:48.605+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:48.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:48.606+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:49.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:49.622+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:49.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:49.622+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:49.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:49.622+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:49.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:49.623+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:49.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:49.732+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:49.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:49.732+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:49.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:49.732+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:49.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:49.733+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:49.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:49.483+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:49.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:49.483+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:49.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:49.483+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:49.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:49.483+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:50.754 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:50.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:50.754 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:50.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:50.755 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:50.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:50.755 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:50.467+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:50.755 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:50.640+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:50.755 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:50.640+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:50.755 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:50.640+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:50.755 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:50.641+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:51.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:50.755+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:51.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:50.755+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:51.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:50.755+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:51.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:50.757+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:51.722 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:51.608+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:51.723 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:51.608+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:51.723 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:51.608+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:51.723 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:51.609+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:51.723 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:51.436+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:51.723 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:51.436+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:51.723 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:51.436+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:51.723 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:51.437+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:52.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:51.724+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:52.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:51.724+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:52.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:51.724+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:52.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:51.725+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:52.752 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:52.469+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:52.752 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:52.469+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:52.752 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:52.469+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:52.752 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:52.469+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:52.753 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:52.600+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:52.753 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:52.600+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:52.753 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:52.600+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:52.753 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:52.601+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:53.140 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:52.753+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:53.140 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:52.753+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:53.140 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:52.753+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:53.141 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:52.754+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.558+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:16:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.558+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:16:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.558+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400f2c0 tx=0x7fcff0013980 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:16:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.558+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400f2c0 tx=0x7fcff0013980 comp rx=0 tx=0).stop 2024-04-27T04:16:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.558+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:16:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.558+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.558+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.558+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.558+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ea0 con 0x7fcfe40447b0 2024-04-27T04:16:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.558+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052fe0 con 0x7fcff8101eb0 2024-04-27T04:16:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.559+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:16:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.560+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:16:53.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.560+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:53.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.560+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4053120 con 0x7fcfe40447b0 2024-04-27T04:16:53.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.560+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fcff003aa10 tx=0x7fcff00519b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:16:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.562+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00103a0 con 0x7fcfe40447b0 2024-04-27T04:16:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.562+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff00195c0 con 0x7fcfe40447b0 2024-04-27T04:16:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:53.562+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059040 con 0x7fcfe40447b0 2024-04-27T04:16:53.746 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:16:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:16:53.559+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6659 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:16:53.746 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:53.442+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:53.746 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:53.442+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:53.747 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:53.442+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:53.747 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:53.442+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:53.747 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:53.567+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:53.747 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:53.567+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:53.747 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:53.567+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:53.747 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:53.567+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:54.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:53.747+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:54.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:53.747+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:54.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:53.747+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:54.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:53.749+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:54.727 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:54.590+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:54.727 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:54.590+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:54.727 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:54.590+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:54.728 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:54.591+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:54.728 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:54.478+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:54.728 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:54.478+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:54.728 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:54.478+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:54.728 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:54.479+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:55.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:54.728+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:55.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:54.728+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:55.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:54.728+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:55.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:54.729+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:55.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:55.698+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:55.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:55.698+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:55.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:55.698+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:55.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:55.699+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:55.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:55.498+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:55.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:55.498+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:55.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:55.498+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:55.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:55.499+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:55.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:55.631+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:55.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:55.631+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:55.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:55.631+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:55.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:55.632+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:56.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:56.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:56.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:56.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:56.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:56.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:56.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:56.490+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:56.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:56.624+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:56.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:56.624+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:56.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:56.624+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:56.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:56.625+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:56.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:56.676+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:56.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:56.676+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:56.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:56.676+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:56.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:56.677+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:57.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:57.699+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:57.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:57.699+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:57.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:57.699+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:57.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:57.700+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:57.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:57.452+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:57.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:57.452+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:57.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:57.452+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:57.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:57.453+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:57.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:57.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:57.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:57.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:57.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:57.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:57.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:57.628+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.559+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:16:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.559+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:16:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.559+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fcff003aa10 tx=0x7fcff00519b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:16:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.559+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fcff003aa10 tx=0x7fcff00519b0 comp rx=0 tx=0).stop 2024-04-27T04:16:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.559+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:16:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.559+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.559+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.559+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:16:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.559+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ea0 con 0x7fcfe4054300 2024-04-27T04:16:58.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.559+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052fe0 con 0x7fcff8101eb0 2024-04-27T04:16:58.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.559+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:16:58.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.560+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:16:58.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.560+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:16:58.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.560+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4053120 con 0x7fcfe4054300 2024-04-27T04:16:58.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.560+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fcff0014fd0 tx=0x7fcff001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:16:58.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.561+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4054300 2024-04-27T04:16:58.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.561+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0037ae0 con 0x7fcfe4054300 2024-04-27T04:16:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:16:58.562+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059040 con 0x7fcfe4054300 2024-04-27T04:16:58.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:16:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:16:58.560+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6722 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:16:58.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:58.404+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:58.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:58.404+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:58.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:58.404+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:58.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:58.405+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:58.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:58.642+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:58.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:58.642+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:58.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:58.642+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:16:58.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:58.643+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:16:59.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:58.745+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:59.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:58.745+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:59.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:58.745+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:16:59.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:58.745+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:16:59.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:59.448+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:59.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:59.448+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:59.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:59.448+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:16:59.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:16:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:16:59.449+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:00.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:59.683+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:00.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:59.683+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:00.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:59.683+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:00.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:16:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:16:59.684+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:00.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:59.696+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:00.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:59.696+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:00.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:59.696+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:00.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:16:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:16:59.697+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:00.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:00.717+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:00.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:00.717+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:00.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:00.717+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:00.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:00.718+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:00.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:00.497+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:00.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:00.497+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:00.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:00.497+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:00.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:00.497+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:00.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:00.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:00.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:00.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:00.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:00.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:00.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:00.708+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:01.823 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:01.520+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:01.823 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:01.520+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:01.823 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:01.520+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:01.823 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:01.520+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:01.824 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:01.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:01.824 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:01.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:01.824 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:01.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:01.824 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:01.758+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:01.824 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:01.707+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:01.824 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:01.707+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:01.824 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:01.707+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:01.824 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:01.708+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:02.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:02.357+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:17:02.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:02.357+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:17:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:02.514+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:02.514+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:02.514+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:02.515+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:03.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:02.801+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:03.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:02.801+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:03.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:02.802+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:03.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:02.803+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:03.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:02.707+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:03.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:02.707+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:03.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:02.707+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:03.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:02.708+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:03.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.560+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:17:03.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.560+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:17:03.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.560+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fcff0014fd0 tx=0x7fcff001b000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:17:03.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.560+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fcff0014fd0 tx=0x7fcff001b000 comp rx=0 tx=0).stop 2024-04-27T04:17:03.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.560+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:17:03.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.560+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:03.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.560+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:03.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.560+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:03.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.560+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052f30 con 0x7fcfe40447b0 2024-04-27T04:17:03.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.560+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4053070 con 0x7fcff8101eb0 2024-04-27T04:17:03.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.561+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:17:03.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.562+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:17:03.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.562+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:03.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.562+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40531b0 con 0x7fcfe40447b0 2024-04-27T04:17:03.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.562+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80138b0 tx=0x7fcff0011c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:17:03.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.563+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe40447b0 2024-04-27T04:17:03.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.563+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe801e970 con 0x7fcfe40447b0 2024-04-27T04:17:03.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:03.564+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe804d9d0 con 0x7fcfe40447b0 2024-04-27T04:17:03.774 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:17:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:17:03.561+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6783 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:17:03.775 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:03.508+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:03.775 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:03.508+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:03.775 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:03.508+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:03.775 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:03.509+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:03.775 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:03.678+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:03.775 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:03.678+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:03.776 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:03.678+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:03.776 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:03.678+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:04.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:03.776+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:04.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:03.776+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:04.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:03.776+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:04.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:03.777+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:04.768 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:04.481+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:04.769 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:04.481+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:04.769 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:04.481+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:04.769 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:04.482+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:04.769 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:04.719+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:04.769 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:04.719+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:04.769 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:04.719+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:04.769 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:04.720+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:05.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:04.770+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:05.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:04.770+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:05.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:04.770+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:05.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:04.771+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:05.698 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:05.527+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:05.699 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:05.527+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:05.699 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:05.527+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:05.699 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:05.528+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:05.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:05.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:05.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:05.758+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:06.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:05.700+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:06.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:05.700+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:06.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:05.700+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:05.700+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:06.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:06.515+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:06.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:06.515+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:06.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:06.515+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:06.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:06.515+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:06.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:06.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:06.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:06.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:06.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:06.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:06.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:06.722+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:06.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:06.671+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:06.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:06.671+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:06.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:06.671+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:06.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:06.672+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:07.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:07.710+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:07.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:07.710+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:07.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:07.710+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:07.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:07.711+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:07.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:07.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:07.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:07.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:07.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:07.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:07.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:07.490+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:07.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:07.733+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:07.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:07.733+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:07.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:07.733+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:07.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:07.734+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.561+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:17:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.561+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:17:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.561+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80138b0 tx=0x7fcff0011c70 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:17:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.561+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80138b0 tx=0x7fcff0011c70 comp rx=0 tx=0).stop 2024-04-27T04:17:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.561+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:17:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.561+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.561+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.561+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.561+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052f30 con 0x7fcfe4054300 2024-04-27T04:17:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.561+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4053070 con 0x7fcff8101eb0 2024-04-27T04:17:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.562+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:17:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.562+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:17:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.562+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.562+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40531b0 con 0x7fcfe4054300 2024-04-27T04:17:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.562+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fcff003a010 tx=0x7fcfe8021fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:17:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.563+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4054300 2024-04-27T04:17:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.563+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe4054300 2024-04-27T04:17:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:08.564+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059040 con 0x7fcfe4054300 2024-04-27T04:17:08.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:17:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:17:08.562+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6846 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:17:08.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:08.709+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:08.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:08.709+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:08.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:08.709+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:08.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:08.709+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:08.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:08.530+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:08.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:08.530+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:08.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:08.530+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:08.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:08.530+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:08.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:08.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:08.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:08.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:08.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:08.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:08.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:08.774+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:09.806 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:09.554+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:09.807 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:09.554+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:09.807 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:09.554+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:09.807 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:09.555+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:09.807 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:09.668+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:09.807 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:09.668+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:09.807 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:09.668+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:09.807 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:09.669+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:09.808+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:09.808+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:09.808+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:09.809+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:10.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:10.824+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:10.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:10.824+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:10.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:10.824+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:10.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:10.825+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:10.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:10.593+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:10.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:10.593+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:10.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:10.593+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:10.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:10.595+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:10.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:10.654+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:10.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:10.654+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:10.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:10.654+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:10.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:10.654+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:11.834 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:11.613+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:11.834 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:11.613+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:11.834 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:11.613+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:11.834 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:11.613+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:11.835 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:11.556+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:11.835 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:11.556+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:11.835 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:11.556+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:11.835 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:11.557+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:12.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:11.835+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:12.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:11.835+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:12.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:11.835+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:12.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:11.836+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:12.830 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:12.513+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:12.830 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:12.513+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:12.830 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:12.513+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:12.830 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:12.514+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:12.830 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:12.612+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:12.830 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:12.612+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:12.831 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:12.612+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:12.831 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:12.612+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:13.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:12.830+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:13.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:12.830+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:13.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:12.830+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:13.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:12.831+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:13.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.561+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:17:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.561+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:17:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.561+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fcff003a010 tx=0x7fcfe8021fb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:17:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.561+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fcff003a010 tx=0x7fcfe8021fb0 comp rx=0 tx=0).stop 2024-04-27T04:17:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.561+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:17:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.561+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.561+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.561+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:13.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.561+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ea0 con 0x7fcfe40447b0 2024-04-27T04:17:13.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.561+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052fe0 con 0x7fcff8101eb0 2024-04-27T04:17:13.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.561+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:17:13.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.562+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:17:13.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.562+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:13.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.562+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4053120 con 0x7fcfe40447b0 2024-04-27T04:17:13.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.562+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fcff0042170 tx=0x7fcff0001530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:17:13.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.562+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe40447b0 2024-04-27T04:17:13.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.563+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe801e970 con 0x7fcfe40447b0 2024-04-27T04:17:13.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:13.563+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe801cd40 con 0x7fcfe40447b0 2024-04-27T04:17:13.787 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:17:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:17:13.561+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6909 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:17:13.787 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:13.581+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:13.787 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:13.581+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:13.788 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:13.581+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:13.788 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:13.582+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:13.788 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:13.532+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:13.788 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:13.532+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:13.788 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:13.532+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:13.788 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:13.533+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:14.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:13.787+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:14.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:13.787+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:14.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:13.787+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:14.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:13.788+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:14.831 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:14.617+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:14.831 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:14.617+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:14.831 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:14.618+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:14.831 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:14.618+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:14.832 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:14.536+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:14.832 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:14.536+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:14.832 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:14.536+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:14.832 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:14.538+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:15.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:14.830+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:15.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:14.831+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:15.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:14.831+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:15.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:14.832+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:15.864 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:15.576+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:15.864 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:15.576+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:15.864 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:15.576+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:15.864 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:15.577+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:15.864 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:15.617+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:15.864 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:15.617+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:15.864 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:15.617+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:15.865 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:15.617+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:16.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:15.864+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:16.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:15.864+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:16.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:15.864+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:16.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:15.865+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:16.908 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:16.573+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:16.908 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:16.573+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:16.908 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:16.573+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:16.908 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:16.574+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:16.909 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:16.583+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:16.909 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:16.583+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:16.909 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:16.583+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:16.909 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:16.584+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:17.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:16.908+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:17.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:16.908+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:17.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:16.908+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:17.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:16.909+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:17.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:17.716+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:17:17.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:17.716+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:17:17.881 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:17.618+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:17.881 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:17.618+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:17.881 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:17.618+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:17.881 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:17.619+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:17.881 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:17.590+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:17.881 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:17.590+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:17.881 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:17.590+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:17.881 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:17.591+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:18.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:17.881+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:18.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:17.881+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:18.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:17.881+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:18.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:17.882+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:18.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.562+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:17:18.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.562+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:17:18.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.562+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fcff0042170 tx=0x7fcff0001530 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:17:18.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.562+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fcff0042170 tx=0x7fcff0001530 comp rx=0 tx=0).stop 2024-04-27T04:17:18.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.562+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:17:18.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.562+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:18.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.562+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:18.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.562+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:18.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.562+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052990 con 0x7fcfe4054300 2024-04-27T04:17:18.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.562+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ea0 con 0x7fcff8101eb0 2024-04-27T04:17:18.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.563+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:17:18.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.563+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:17:18.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.563+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:18.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.563+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052fe0 con 0x7fcfe4054300 2024-04-27T04:17:18.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.563+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8016ef0 tx=0x7fcfe802fde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:17:18.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.564+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4054300 2024-04-27T04:17:18.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.564+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8042330 con 0x7fcfe4054300 2024-04-27T04:17:18.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:18.566+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe801cd40 con 0x7fcfe4054300 2024-04-27T04:17:18.876 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:17:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:17:18.563+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 6973 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:17:18.877 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:18.582+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:18.877 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:18.582+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:18.877 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:18.582+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:18.877 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:18.583+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:18.877 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:18.549+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:18.877 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:18.549+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:18.877 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:18.549+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:18.877 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:18.549+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:18.878 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:18.877+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:19.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:18.877+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:19.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:18.877+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:19.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:18.878+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:19.865 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:19.583+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:19.865 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:19.583+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:19.865 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:19.583+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:19.866 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:19.584+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:19.866 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:19.632+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:19.866 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:19.632+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:19.866 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:19.632+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:19.866 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:19.633+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:20.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:19.866+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:20.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:19.866+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:20.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:19.866+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:20.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:19.867+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:20.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:20.833+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:20.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:20.833+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:20.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:20.833+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:20.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:20.834+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:20.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:20.586+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:20.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:20.586+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:20.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:20.586+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:20.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:20.587+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:20.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:20.669+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:20.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:20.670+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:20.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:20.670+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:20.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:20.671+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:21.818 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:21.582+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:21.818 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:21.582+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:21.819 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:21.582+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:21.819 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:21.582+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:21.819 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:21.674+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:21.819 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:21.674+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:21.819 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:21.674+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:21.819 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:21.675+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:22.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:21.819+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:22.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:21.819+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:22.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:21.819+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:22.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:21.820+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:22.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:22.625+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:22.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:22.625+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:22.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:22.625+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:22.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:22.627+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:22.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:22.781+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:22.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:22.781+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:22.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:22.781+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:22.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:22.782+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:22.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:22.591+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:22.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:22.591+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:22.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:22.591+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:22.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:22.592+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.563+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:17:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.563+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:17:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.563+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8016ef0 tx=0x7fcfe802fde0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:17:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.563+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8016ef0 tx=0x7fcfe802fde0 comp rx=0 tx=0).stop 2024-04-27T04:17:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.563+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:17:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.563+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.563+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.563+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.563+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404fd00 con 0x7fcfe40447b0 2024-04-27T04:17:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.563+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ea0 con 0x7fcff8101eb0 2024-04-27T04:17:23.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.564+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:17:23.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.565+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:17:23.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.565+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:23.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.565+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052fe0 con 0x7fcfe40447b0 2024-04-27T04:17:23.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.565+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8032f20 tx=0x7fcfe803cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:17:23.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.565+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe40447b0 2024-04-27T04:17:23.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.565+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe40447b0 2024-04-27T04:17:23.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:23.567+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe801cd40 con 0x7fcfe40447b0 2024-04-27T04:17:23.859 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:17:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:17:23.564+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 7035 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:17:23.860 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:23.569+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:23.860 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:23.569+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:23.860 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:23.569+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:23.860 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:23.570+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:23.860 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:23.618+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:23.861 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:23.618+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:23.861 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:23.618+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:23.861 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:23.619+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:23.861 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:23.806+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:23.861 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:23.806+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:23.861 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:23.806+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:23.861 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:23.806+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:24.820 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:24.669+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:24.820 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:24.669+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:24.820 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:24.669+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:24.820 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:24.669+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:24.820 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:24.521+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:24.820 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:24.521+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:24.820 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:24.521+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:24.820 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:24.522+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:25.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:24.820+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:25.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:24.820+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:25.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:24.820+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:25.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:24.821+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:25.806 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:25.488+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:25.806 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:25.488+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:25.807 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:25.488+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:25.807 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:25.488+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:25.807 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:25.670+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:25.807 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:25.670+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:25.807 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:25.670+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:25.807 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:25.672+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:26.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:25.808+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:26.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:25.808+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:26.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:25.808+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:26.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:25.808+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:26.798 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:26.655+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:26.799 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:26.655+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:26.799 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:26.655+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:26.799 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:26.656+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:26.799 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:26.509+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:26.799 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:26.510+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:26.799 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:26.510+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:26.799 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:26.511+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:27.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:26.800+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:27.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:26.800+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:27.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:26.800+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:27.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:26.801+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:27.812 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:27.675+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:27.812 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:27.675+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:27.812 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:27.675+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:27.812 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:27.676+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:27.812 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:27.547+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:27.812 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:27.547+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:27.812 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:27.547+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:27.812 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:27.548+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:28.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:27.813+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:28.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:27.813+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:28.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:27.813+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:28.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:27.814+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:28.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.565+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:17:28.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.565+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:17:28.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.565+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8032f20 tx=0x7fcfe803cc70 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:17:28.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.565+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8032f20 tx=0x7fcfe803cc70 comp rx=0 tx=0).stop 2024-04-27T04:17:28.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.565+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:17:28.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.565+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:28.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.565+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:28.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.565+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:28.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.565+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404fd00 con 0x7fcfe4054300 2024-04-27T04:17:28.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.565+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ea0 con 0x7fcff8101eb0 2024-04-27T04:17:28.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.566+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:17:28.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.566+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:17:28.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.566+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:28.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.566+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052fe0 con 0x7fcfe4054300 2024-04-27T04:17:28.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.567+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fcff004b010 tx=0x7fcfe804fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:17:28.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.568+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4054300 2024-04-27T04:17:28.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.569+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe4054300 2024-04-27T04:17:28.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:28.569+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4054300 2024-04-27T04:17:28.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:17:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:17:28.566+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 7098 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:17:28.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:28.585+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:28.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:28.585+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:28.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:28.585+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:28.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:28.586+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:28.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:28.810+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:28.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:28.810+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:28.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:28.810+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:28.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:28.811+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:28.704+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:28.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:28.704+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:28.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:28.704+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:28.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:28.704+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:29.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:29.735+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:29.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:29.735+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:29.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:29.735+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:29.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:29.736+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:29.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:29.833+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:29.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:29.833+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:29.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:29.833+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:29.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:29.833+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:29.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:29.609+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:29.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:29.609+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:29.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:29.609+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:29.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:29.609+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:30.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:30.630+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:30.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:30.630+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:30.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:30.630+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:30.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:30.631+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:30.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:30.766+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:30.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:30.766+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:30.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:30.766+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:30.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:30.767+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:30.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:30.819+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:30.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:30.819+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:30.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:30.819+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:30.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:30.820+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:31.822 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:31.815+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:31.822 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:31.815+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:31.823 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:31.815+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:31.823 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:31.815+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:31.823 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:31.780+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:31.823 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:31.780+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:31.823 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:31.780+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:31.823 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:31.780+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:31.823 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:31.602+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:31.824 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:31.602+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:31.824 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:31.602+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:31.824 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:31.602+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:32.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:32.572+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:32.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:32.572+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:32.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:32.572+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:32.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:32.572+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:32.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:32.745+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:32.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:32.745+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:32.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:32.745+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:32.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:32.745+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:32.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:32.806+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:32.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:32.806+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:32.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:32.806+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:32.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:32.807+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.141+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:17:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.141+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:17:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.566+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:17:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.566+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:17:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.566+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fcff004b010 tx=0x7fcfe804fa10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:17:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.566+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fcff004b010 tx=0x7fcfe804fa10 comp rx=0 tx=0).stop 2024-04-27T04:17:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.566+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:17:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.566+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.566+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.566+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.566+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ea0 con 0x7fcfe40447b0 2024-04-27T04:17:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.566+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052fe0 con 0x7fcff8101eb0 2024-04-27T04:17:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.566+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:17:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.567+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:17:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.567+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.567+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4053120 con 0x7fcfe40447b0 2024-04-27T04:17:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.567+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fcfe801ba80 tx=0x7fcff004df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:17:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.569+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe40447b0 2024-04-27T04:17:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.569+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe801e970 con 0x7fcfe40447b0 2024-04-27T04:17:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:33.570+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe801cd40 con 0x7fcfe40447b0 2024-04-27T04:17:33.710 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:17:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:17:33.567+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 7161 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:17:33.710 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:33.523+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:33.710 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:33.523+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:33.710 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:33.523+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:33.710 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:33.524+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:34.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:33.844+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:34.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:33.844+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:34.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:33.844+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:34.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:33.845+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:34.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:33.711+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:34.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:33.711+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:34.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:33.711+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:34.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:33.712+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:34.748 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:34.496+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:34.749 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:34.496+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:34.749 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:34.496+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:34.749 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:34.497+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:35.122 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:34.862+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:35.122 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:34.862+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:35.122 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:34.862+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:35.122 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:34.863+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:35.122 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:34.750+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:35.122 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:34.750+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:35.122 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:34.750+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:35.122 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:34.751+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:35.761 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:35.470+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:35.761 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:35.470+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:35.761 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:35.470+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:35.761 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:35.470+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:36.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:35.871+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:36.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:35.871+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:36.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:35.871+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:36.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:35.872+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:36.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:35.762+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:36.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:35.762+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:36.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:35.762+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:36.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:35.764+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:36.891 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:36.749+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:36.891 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:36.749+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:36.892 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:36.749+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:36.892 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:36.751+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:36.892 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:36.511+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:36.892 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:36.511+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:36.892 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:36.511+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:36.892 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:36.512+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:37.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:36.892+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:37.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:36.892+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:37.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:36.892+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:37.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:36.894+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:37.847 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:37.486+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:37.848 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:37.486+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:37.848 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:37.486+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:37.848 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:37.487+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:37.848 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:37.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:37.848 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:37.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:37.848 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:37.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:37.848 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:37.716+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:38.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:37.849+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:38.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:37.849+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:38.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:37.849+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:38.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:37.850+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:38.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.567+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:17:38.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.567+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:17:38.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.567+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fcfe801ba80 tx=0x7fcff004df30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:17:38.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.567+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fcfe801ba80 tx=0x7fcff004df30 comp rx=0 tx=0).stop 2024-04-27T04:17:38.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.567+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:17:38.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.567+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:38.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.567+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:38.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.567+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:38.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.567+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ea0 con 0x7fcfe4054300 2024-04-27T04:17:38.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.567+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052fe0 con 0x7fcff8101eb0 2024-04-27T04:17:38.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.567+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:17:38.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.568+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:17:38.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.568+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:38.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.568+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4053120 con 0x7fcfe4054300 2024-04-27T04:17:38.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.568+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fcff002a210 tx=0x7fcfe800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:17:38.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.570+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4054300 2024-04-27T04:17:38.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.570+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff001a6a0 con 0x7fcfe4054300 2024-04-27T04:17:38.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:38.570+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe4054300 2024-04-27T04:17:38.712 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:17:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:17:38.567+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 7225 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:17:38.713 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:38.457+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:38.713 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:38.457+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:38.713 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:38.457+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:38.713 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:38.457+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:39.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:38.714+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:39.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:38.714+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:39.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:38.714+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:39.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:38.715+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:39.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:38.878+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:39.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:38.878+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:39.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:38.878+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:39.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:38.879+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:39.667 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:39.431+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:39.668 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:39.431+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:39.668 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:39.431+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:39.668 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:39.431+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:39.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:39.669+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:39.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:39.669+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:39.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:39.669+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:39.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:39.670+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:39.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:39.830+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:39.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:39.830+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:39.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:39.830+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:39.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:39.832+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:40.822 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:40.663+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:40.822 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:40.663+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:40.822 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:40.663+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:40.822 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:40.664+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:40.822 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:40.462+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:40.823 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:40.462+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:40.823 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:40.462+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:40.823 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:40.462+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:41.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:40.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:41.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:40.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:41.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:40.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:41.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:40.824+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:41.781 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:41.619+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:41.782 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:41.619+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:41.782 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:41.619+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:41.782 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:41.621+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:41.782 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:41.498+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:41.782 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:41.498+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:41.782 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:41.498+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:41.782 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:41.499+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:42.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:41.783+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:42.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:41.783+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:42.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:41.783+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:42.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:41.784+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:42.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:42.601+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:42.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:42.601+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:42.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:42.601+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:42.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:42.602+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:42.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:42.465+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:42.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:42.465+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:42.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:42.465+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:42.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:42.466+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:43.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:42.779+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:43.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:42.779+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:43.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:42.779+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:43.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:42.780+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:43.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.568+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:17:43.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.568+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:17:43.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.568+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fcff002a210 tx=0x7fcfe800bda0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:17:43.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.568+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fcff002a210 tx=0x7fcfe800bda0 comp rx=0 tx=0).stop 2024-04-27T04:17:43.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.568+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:17:43.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.568+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:43.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.568+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:43.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.568+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:43.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.568+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ea0 con 0x7fcfe40447b0 2024-04-27T04:17:43.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.568+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052fe0 con 0x7fcff8101eb0 2024-04-27T04:17:43.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.568+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:17:43.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.569+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:17:43.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.569+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:43.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.569+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4053120 con 0x7fcfe40447b0 2024-04-27T04:17:43.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.570+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4037940 tx=0x7fcff000e450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:17:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.572+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe801e970 con 0x7fcfe40447b0 2024-04-27T04:17:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.572+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037900 con 0x7fcfe40447b0 2024-04-27T04:17:43.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:43.573+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe801cd40 con 0x7fcfe40447b0 2024-04-27T04:17:43.799 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:17:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:17:43.568+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 7287 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:17:43.799 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:43.591+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:43.800 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:43.591+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:43.800 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:43.591+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:43.800 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:43.591+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:43.800 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:43.481+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:43.800 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:43.481+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:43.800 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:43.481+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:43.800 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:43.482+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:44.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:43.801+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:44.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:43.801+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:44.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:43.801+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:44.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:43.802+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:44.802 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:44.487+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:44.802 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:44.487+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:44.802 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:44.487+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:44.802 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:44.488+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:44.803 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:44.584+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:44.803 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:44.584+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:44.803 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:44.584+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:44.803 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:44.585+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:45.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:44.803+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:45.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:44.803+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:45.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:44.803+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:45.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:44.804+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:45.818 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:45.538+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:45.819 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:45.538+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:45.819 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:45.538+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:45.819 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:45.540+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:45.819 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:45.514+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:45.819 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:45.514+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:45.819 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:45.514+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:45.819 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:45.514+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:46.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:45.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:46.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:45.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:46.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:45.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:46.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:45.821+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:46.803 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:46.529+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:46.803 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:46.529+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:46.803 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:46.529+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:46.803 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:46.529+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:46.804 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:46.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:46.804 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:46.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:46.804 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:46.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:46.804 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:46.542+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:47.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:46.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:47.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:46.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:47.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:46.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:47.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:46.806+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:47.834 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:47.526+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:47.835 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:47.526+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:47.835 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:47.526+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:47.835 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:47.526+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:47.835 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:47.510+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:47.835 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:47.510+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:47.835 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:47.510+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:47.835 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:47.511+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:48.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:47.836+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:48.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:47.836+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:48.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:47.836+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:48.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:47.837+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.501+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:17:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.501+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:17:48.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.569+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:17:48.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.569+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:17:48.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.569+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4037940 tx=0x7fcff000e450 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:17:48.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.569+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4037940 tx=0x7fcff000e450 comp rx=0 tx=0).stop 2024-04-27T04:17:48.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.569+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:17:48.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.569+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:48.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.569+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:48.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.569+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:48.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.569+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ea0 con 0x7fcfe4054300 2024-04-27T04:17:48.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.569+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052fe0 con 0x7fcff8101eb0 2024-04-27T04:17:48.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.569+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:17:48.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.570+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:17:48.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.570+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:48.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.570+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4053120 con 0x7fcfe4054300 2024-04-27T04:17:48.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.571+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fcff0042fc0 tx=0x7fcfe8026940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:17:48.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.572+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff00092c0 con 0x7fcfe4054300 2024-04-27T04:17:48.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.572+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe4054300 2024-04-27T04:17:48.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:48.572+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff003c110 con 0x7fcfe4054300 2024-04-27T04:17:48.861 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:17:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:17:48.569+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 7350 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:17:48.862 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:48.535+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:48.862 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:48.535+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:48.862 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:48.535+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:48.862 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:48.536+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:48.863 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:48.541+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:48.863 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:48.541+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:48.863 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:48.541+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:48.863 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:48.541+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:49.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:48.863+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:49.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:48.863+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:49.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:48.863+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:49.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:48.864+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:49.912 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:49.532+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:49.912 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:49.532+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:49.912 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:49.532+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:49.912 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:49.533+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:49.913 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:49.555+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:49.913 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:49.555+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:49.913 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:49.555+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:49.913 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:49.556+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:50.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:49.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:50.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:49.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:50.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:49.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:50.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:49.915+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:50.551 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:50.522+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:50.552 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:50.522+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:50.552 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:50.522+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:50.552 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:50.522+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:50.911 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:50.553+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:50.911 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:50.553+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:50.911 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:50.553+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:50.911 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:50.554+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:51.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:50.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:51.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:50.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:51.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:50.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:51.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:50.913+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:51.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:51.515+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:51.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:51.515+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:51.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:51.515+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:51.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:51.515+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:51.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:51.495+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:51.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:51.495+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:51.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:51.495+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:51.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:51.495+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:52.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:51.947+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:52.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:51.947+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:52.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:51.947+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:52.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:51.948+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:52.916 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:52.467+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:52.916 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:52.467+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:52.916 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:52.467+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:52.917 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:52.468+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:52.917 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:52.547+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:52.917 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:52.547+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:52.917 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:52.547+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:52.917 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:52.547+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:53.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:52.917+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:53.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:52.917+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:53.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:52.917+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:53.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:52.919+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.570+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:17:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.570+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:17:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.570+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fcff0042fc0 tx=0x7fcfe8026940 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:17:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.570+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fcff0042fc0 tx=0x7fcfe8026940 comp rx=0 tx=0).stop 2024-04-27T04:17:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.570+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:17:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.570+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.570+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.570+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.570+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ea0 con 0x7fcfe40447b0 2024-04-27T04:17:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.570+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052fe0 con 0x7fcff8101eb0 2024-04-27T04:17:53.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.570+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:17:53.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.571+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:17:53.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.571+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:53.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.571+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4053120 con 0x7fcfe40447b0 2024-04-27T04:17:53.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.571+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4052cd0 tx=0x7fcff000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:17:53.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.572+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe40447b0 2024-04-27T04:17:53.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.572+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0003740 con 0x7fcfe40447b0 2024-04-27T04:17:53.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:53.573+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff003c110 con 0x7fcfe40447b0 2024-04-27T04:17:53.911 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:17:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:17:53.570+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 7413 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:17:53.911 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:53.529+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:53.911 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:53.529+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:53.911 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:53.529+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:53.911 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:53.530+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:53.911 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:53.459+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:53.912 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:53.459+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:53.912 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:53.459+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:53.912 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:53.460+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:54.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:53.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:54.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:53.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:54.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:53.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:54.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:53.913+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:54.876 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:54.572+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:54.876 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:54.572+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:54.876 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:54.572+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:54.876 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:54.573+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:54.876 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:54.484+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:54.876 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:54.484+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:54.876 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:54.484+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:54.877 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:54.484+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:55.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:54.877+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:55.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:54.877+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:55.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:54.877+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:55.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:54.879+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:55.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:55.508+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:55.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:55.508+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:55.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:55.508+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:55.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:55.509+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:55.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:55.587+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:55.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:55.587+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:55.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:55.587+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:55.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:55.588+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:56.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:55.883+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:56.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:55.883+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:56.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:55.883+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:56.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:55.884+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:56.890 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:56.498+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:56.890 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:56.498+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:56.890 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:56.498+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:56.890 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:56.499+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:56.891 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:56.538+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:56.891 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:56.538+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:56.891 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:56.538+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:56.891 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:56.539+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:57.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:56.891+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:57.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:56.891+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:57.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:56.891+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:57.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:56.892+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:57.912 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:57.542+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:57.912 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:57.542+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:57.912 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:57.542+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:57.912 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:57.543+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:57.912 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:57.521+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:57.912 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:57.521+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:57.912 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:57.521+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:57.912 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:57.522+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:58.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:57.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:58.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:57.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:58.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:57.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:58.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:57.914+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.571+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:17:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.571+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:17:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.571+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4052cd0 tx=0x7fcff000ee50 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:17:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.571+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4052cd0 tx=0x7fcff000ee50 comp rx=0 tx=0).stop 2024-04-27T04:17:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.571+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:17:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.571+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.571+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.571+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:17:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.571+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4043c90 con 0x7fcfe4054300 2024-04-27T04:17:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.571+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052ea0 con 0x7fcff8101eb0 2024-04-27T04:17:58.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.571+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:17:58.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.572+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:17:58.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.572+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:17:58.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.572+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4052fe0 con 0x7fcfe4054300 2024-04-27T04:17:58.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.572+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4018490 tx=0x7fcff0006230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:17:58.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.573+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe801e970 con 0x7fcfe4054300 2024-04-27T04:17:58.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.573+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4054300 2024-04-27T04:17:58.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:17:58.574+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013a90 con 0x7fcfe4054300 2024-04-27T04:17:58.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:17:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:17:58.571+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 7477 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:17:58.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:58.552+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:58.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:58.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:58.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:58.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:58.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:58.553+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:17:58.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:58.525+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:58.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:58.525+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:58.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:58.525+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:58.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:58.526+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:59.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:58.938+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:59.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:58.938+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:59.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:58.938+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:17:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:58.939+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:59.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:59.483+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:59.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:59.483+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:59.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:59.483+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:17:59.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:17:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:17:59.484+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:17:59.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:59.514+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:59.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:59.514+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:59.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:59.514+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:17:59.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:17:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:17:59.515+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:00.399 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:59.959+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:00.399 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:59.959+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:00.400 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:59.959+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:00.400 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:17:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:17:59.960+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:00.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:00.467+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:00.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:00.467+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:00.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:00.467+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:00.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:00.468+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:00.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:00.507+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:00.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:00.507+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:00.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:00.507+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:00.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:00.508+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:01.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:00.995+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:01.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:00.995+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:01.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:00.995+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:01.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:00.996+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:01.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:01.547+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:01.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:01.547+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:01.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:01.547+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:01.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:01.548+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:01.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:01.479+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:01.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:01.479+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:01.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:01.479+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:01.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:01.479+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:02.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:02.003+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:02.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:02.003+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:02.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:02.003+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:02.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:02.003+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:02.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:02.551+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:02.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:02.551+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:02.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:02.551+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:02.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:02.551+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:02.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:02.505+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:02.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:02.505+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:02.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:02.505+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:02.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:02.506+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:03.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:02.995+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:03.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:02.995+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:03.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:02.995+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:03.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:02.996+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.572+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:18:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.572+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:18:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.572+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4018490 tx=0x7fcff0006230 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:18:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.572+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4018490 tx=0x7fcff0006230 comp rx=0 tx=0).stop 2024-04-27T04:18:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.572+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:18:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.572+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.572+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.572+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.572+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052f70 con 0x7fcfe40447b0 2024-04-27T04:18:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.572+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40530b0 con 0x7fcff8101eb0 2024-04-27T04:18:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.572+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:18:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.573+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:18:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.573+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.573+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40531f0 con 0x7fcfe40447b0 2024-04-27T04:18:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.574+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fcfe802fe20 tx=0x7fcfe800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:18:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.575+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe801e970 con 0x7fcfe40447b0 2024-04-27T04:18:03.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.575+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe40447b0 2024-04-27T04:18:03.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.576+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013a90 con 0x7fcfe40447b0 2024-04-27T04:18:03.576 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:18:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:18:03.572+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 7539 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:18:03.576 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:03.465+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:03.576 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:03.465+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:03.576 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:03.465+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:03.576 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:03.466+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:03.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.861+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:18:03.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:03.861+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:18:03.860 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:03.575+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:03.860 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:03.575+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:03.860 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:03.575+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:03.860 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:03.576+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:04.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:03.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:04.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:03.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:04.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:03.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:04.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:03.974+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:04.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:04.474+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:04.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:04.474+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:04.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:04.474+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:04.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:04.475+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:04.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:04.573+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:04.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:04.573+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:04.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:04.573+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:04.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:04.575+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:05.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:04.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:05.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:04.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:05.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:04.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:05.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:04.976+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:05.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:05.517+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:05.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:05.517+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:05.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:05.517+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:05.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:05.517+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:05.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:05.540+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:05.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:05.540+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:05.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:05.540+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:05.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:05.542+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:06.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:05.930+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:06.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:05.930+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:06.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:05.930+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:06.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:05.930+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:06.895 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:06.531+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:06.895 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:06.531+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:06.895 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:06.531+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:06.895 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:06.532+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:06.895 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:06.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:06.896 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:06.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:06.896 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:06.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:06.896 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:06.493+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:07.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:06.896+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:07.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:06.896+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:07.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:06.896+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:07.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:06.897+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:07.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:07.518+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:07.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:07.518+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:07.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:07.518+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:07.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:07.519+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:07.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:07.547+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:07.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:07.547+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:07.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:07.547+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:07.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:07.548+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:08.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:07.937+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:08.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:07.937+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:08.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:07.937+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:08.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:07.938+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.573+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-27T04:18:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.573+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:18:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.573+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fcfe802fe20 tx=0x7fcfe800bda0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:18:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.573+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fcfe802fe20 tx=0x7fcfe800bda0 comp rx=0 tx=0).stop 2024-04-27T04:18:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.573+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:18:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.573+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.573+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.573+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.573+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052f50 con 0x7fcfe4054300 2024-04-27T04:18:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.573+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4053090 con 0x7fcff8101eb0 2024-04-27T04:18:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.573+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:18:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.574+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:18:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.574+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.574+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40531d0 con 0x7fcfe4054300 2024-04-27T04:18:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.575+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804f020 tx=0x7fcfe8037f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:18:08.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.575+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe801e970 con 0x7fcfe4054300 2024-04-27T04:18:08.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.577+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4054300 2024-04-27T04:18:08.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:08.577+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013a90 con 0x7fcfe4054300 2024-04-27T04:18:08.888 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:18:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:18:08.573+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 7602 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:18:08.889 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:08.535+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:08.889 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:08.535+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:08.889 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:08.535+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:08.889 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:08.536+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:08.889 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:08.518+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:08.889 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:08.518+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:08.889 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:08.518+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:08.889 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:08.519+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:09.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:08.890+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:09.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:08.890+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:09.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:08.890+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:09.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:08.891+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:09.867 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:09.545+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:09.939 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:09.545+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:09.939 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:09.545+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:09.939 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:09.545+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:09.939 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:09.534+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:09.939 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:09.534+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:09.939 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:09.534+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:09.939 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:09.535+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:10.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:09.868+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:10.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:09.868+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:10.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:09.868+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:10.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:09.869+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:10.915 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:10.504+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:10.915 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:10.504+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:10.915 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:10.504+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:10.915 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:10.505+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:10.916 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:10.546+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:10.916 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:10.546+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:10.916 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:10.546+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:10.916 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:10.547+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:11.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:10.916+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:11.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:10.916+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:11.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:10.916+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:11.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:10.917+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:11.904 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:11.489+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:11.905 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:11.489+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:11.905 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:11.489+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:11.905 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:11.489+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:11.905 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:11.590+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:11.905 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:11.590+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:11.905 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:11.590+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:11.905 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:11.590+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:12.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:11.906+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:12.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:11.906+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:12.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:11.906+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:12.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:11.907+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:12.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:12.582+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:12.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:12.582+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:12.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:12.582+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:12.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:12.583+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:12.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:12.497+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:12.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:12.497+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:12.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:12.497+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:12.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:12.497+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:13.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:12.940+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:13.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:12.940+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:13.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:12.940+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:13.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:12.941+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:13.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.574+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:18:13.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.574+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:18:13.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.574+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804f020 tx=0x7fcfe8037f60 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:18:13.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.574+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fcfe804f020 tx=0x7fcfe8037f60 comp rx=0 tx=0).stop 2024-04-27T04:18:13.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.574+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:18:13.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.574+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:13.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.574+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:13.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.574+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:13.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.574+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052f50 con 0x7fcfe40447b0 2024-04-27T04:18:13.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.574+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4053090 con 0x7fcff8101eb0 2024-04-27T04:18:13.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.575+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:18:13.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.575+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:18:13.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.575+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:13.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.575+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40531d0 con 0x7fcfe40447b0 2024-04-27T04:18:13.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.576+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fcff005dfa0 tx=0x7fcfe8033960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:18:13.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.576+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff003a340 con 0x7fcfe40447b0 2024-04-27T04:18:13.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.576+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe40447b0 2024-04-27T04:18:13.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:13.578+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe40447b0 2024-04-27T04:18:13.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:18:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:18:13.575+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 7665 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:18:13.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:13.602+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:13.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:13.602+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:13.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:13.602+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:13.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:13.603+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:13.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:13.504+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:13.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:13.504+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:13.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:13.504+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:13.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:13.505+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:14.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:13.956+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:14.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:13.956+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:14.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:13.956+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:14.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:13.958+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:14.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:14.623+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:14.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:14.623+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:14.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:14.623+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:14.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:14.623+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:14.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:14.512+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:14.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:14.512+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:14.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:14.512+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:14.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:14.512+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:15.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:15.006+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:15.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:15.006+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:15.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:15.006+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:15.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:15.007+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:15.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:15.637+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:15.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:15.637+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:15.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:15.637+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:15.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:15.637+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:15.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:15.474+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:15.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:15.474+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:15.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:15.474+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:15.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:15.475+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:16.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:15.961+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:16.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:15.961+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:16.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:15.961+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:16.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:15.962+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:16.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:16.632+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:16.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:16.632+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:16.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:16.632+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:16.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:16.633+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:16.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:16.469+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:16.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:16.469+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:16.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:16.469+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:16.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:16.470+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:17.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:16.974+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:17.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:16.974+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:17.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:16.974+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:17.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:16.975+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:17.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:17.497+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:17.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:17.497+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:17.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:17.497+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:17.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:17.498+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:17.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:17.625+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:17.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:17.625+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:17.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:17.625+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:17.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:17.626+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:18.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:17.948+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:18.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:17.948+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:18.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:17.948+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:18.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:17.949+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:18.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.575+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:18:18.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.575+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:18:18.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.575+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fcff005dfa0 tx=0x7fcfe8033960 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:18:18.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.575+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fcff005dfa0 tx=0x7fcfe8033960 comp rx=0 tx=0).stop 2024-04-27T04:18:18.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.575+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:18:18.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.576+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:18.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.576+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:18.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.576+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:18.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.576+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4052f50 con 0x7fcfe4054300 2024-04-27T04:18:18.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.576+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4053090 con 0x7fcff8101eb0 2024-04-27T04:18:18.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.576+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:18:18.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.577+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:18:18.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.577+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:18.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.577+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40531d0 con 0x7fcfe4054300 2024-04-27T04:18:18.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.578+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4043b40 tx=0x7fcff0009d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:18:18.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.579+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800cea0 con 0x7fcfe4054300 2024-04-27T04:18:18.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.579+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4054300 2024-04-27T04:18:18.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:18.580+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013a90 con 0x7fcfe4054300 2024-04-27T04:18:18.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:18:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:18:18.576+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 7729 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:18:18.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:18.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:18.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:18.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:18.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:18.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:18.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:18.513+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:18.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:18.630+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:18.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:18.630+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:18.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:18.630+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:18.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:18.630+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:19.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:18.958+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:19.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:18.958+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:19.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:18.958+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:19.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:18.960+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:19.669+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:18:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:19.669+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:18:19.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:19.507+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:19.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:19.507+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:19.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:19.507+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:19.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:19.507+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:19.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:19.679+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:19.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:19.679+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:19.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:19.679+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:19.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:19.681+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:20.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:20.006+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:20.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:20.006+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:20.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:20.006+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:20.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:20.007+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:20.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:20.547+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:20.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:20.547+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:20.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:20.547+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:20.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:20.548+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:20.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:20.681+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:20.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:20.681+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:20.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:20.681+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:20.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:20.682+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:21.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:20.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:21.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:20.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:21.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:20.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:21.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:20.995+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:21.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:21.575+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:21.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:21.575+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:21.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:21.575+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:21.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:21.576+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:21.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:21.655+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:21.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:21.655+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:21.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:21.655+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:21.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:21.656+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:22.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:21.959+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:22.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:21.959+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:22.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:21.959+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:22.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:21.960+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:22.920 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:22.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:22.920 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:22.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:22.920 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:22.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:22.920 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:22.553+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:22.920 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:22.627+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:22.920 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:22.627+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:22.920 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:22.627+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:22.920 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:22.628+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:23.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:22.921+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:23.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:22.921+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:23.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:22.921+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:23.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:22.922+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.577+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:18:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.577+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:18:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.577+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4043b40 tx=0x7fcff0009d60 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:18:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.577+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4043b40 tx=0x7fcff0009d60 comp rx=0 tx=0).stop 2024-04-27T04:18:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.577+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:18:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.577+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.577+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.577+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.577+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4053090 con 0x7fcfe40447b0 2024-04-27T04:18:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.577+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40531d0 con 0x7fcff8101eb0 2024-04-27T04:18:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.577+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:18:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.578+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:18:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.578+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.578+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4053310 con 0x7fcfe40447b0 2024-04-27T04:18:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.578+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fcff004cfb0 tx=0x7fcfe8026360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:18:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.579+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe40447b0 2024-04-27T04:18:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.579+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe40447b0 2024-04-27T04:18:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:23.580+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059070 con 0x7fcfe40447b0 2024-04-27T04:18:23.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:18:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:18:23.578+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 7791 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:18:23.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:23.632+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:23.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:23.632+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:23.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:23.632+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:23.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:23.633+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:23.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:23.597+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:23.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:23.597+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:23.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:23.597+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:23.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:23.598+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:24.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:23.949+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:24.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:23.949+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:24.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:23.949+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:24.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:23.950+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:24.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:24.645+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:24.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:24.645+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:24.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:24.645+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:24.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:24.646+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:24.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:24.613+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:24.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:24.613+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:24.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:24.613+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:24.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:24.614+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:25.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:24.933+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:25.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:24.933+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:25.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:24.933+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:25.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:24.933+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:25.886+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:25.886+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:25.886+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:25.887+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:25.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:25.696+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:25.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:25.696+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:25.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:25.696+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:25.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:25.696+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:25.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:25.664+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:25.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:25.664+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:25.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:25.664+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:25.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:25.664+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:27.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:26.874+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:27.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:26.874+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:27.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:26.874+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:27.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:26.875+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:27.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:26.703+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:27.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:26.703+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:27.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:26.703+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:27.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:26.704+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:27.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:26.694+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:27.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:26.694+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:27.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:26.694+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:27.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:26.695+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:27.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:27.874+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:27.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:27.874+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:27.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:27.874+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:27.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:27.875+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:27.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:27.670+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:27.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:27.670+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:27.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:27.670+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:27.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:27.671+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:27.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:27.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:27.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:27.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:27.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:27.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:27.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:27.687+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:28.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.578+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:18:28.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.578+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:18:28.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.578+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fcff004cfb0 tx=0x7fcfe8026360 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:18:28.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.578+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fcff004cfb0 tx=0x7fcfe8026360 comp rx=0 tx=0).stop 2024-04-27T04:18:28.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.578+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:18:28.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.578+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:28.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.578+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:28.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.578+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:28.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.578+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4053090 con 0x7fcfe4054300 2024-04-27T04:18:28.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.578+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40531d0 con 0x7fcff8101eb0 2024-04-27T04:18:28.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.578+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:18:28.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.579+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:18:28.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.579+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:28.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.579+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4053310 con 0x7fcfe4054300 2024-04-27T04:18:28.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.579+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8010f70 tx=0x7fcff002a9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:18:28.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.580+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800cea0 con 0x7fcfe4054300 2024-04-27T04:18:28.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.580+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe4054300 2024-04-27T04:18:28.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:28.581+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013a90 con 0x7fcfe4054300 2024-04-27T04:18:28.869 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:18:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:18:28.579+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 7854 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:18:28.869 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:28.692+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:28.869 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:28.692+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:28.870 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:28.692+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:28.870 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:28.692+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:28.870 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:28.713+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:28.870 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:28.713+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:28.870 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:28.713+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:28.870 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:28.714+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:29.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:28.870+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:29.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:28.870+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:29.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:28.870+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:29.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:28.871+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:29.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:29.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:29.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:29.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:29.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:29.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:29.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:29.747+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:29.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:29.840+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:29.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:29.840+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:29.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:29.840+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:29.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:29.841+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:29.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:29.646+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:29.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:29.646+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:29.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:29.646+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:29.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:29.647+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:30.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:30.779+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:30.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:30.779+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:30.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:30.779+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:30.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:30.781+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:30.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:30.849+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:30.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:30.849+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:30.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:30.849+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:30.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:30.850+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:30.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:30.650+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:30.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:30.650+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:30.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:30.650+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:30.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:30.650+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:31.822 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:31.667+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:31.823 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:31.667+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:31.823 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:31.667+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:31.823 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:31.668+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:31.823 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:31.751+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:31.823 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:31.751+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:31.823 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:31.751+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:31.823 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:31.752+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:32.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:31.832+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:32.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:31.832+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:32.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:31.832+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:32.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:31.832+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:32.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:32.831+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:32.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:32.831+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:32.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:32.831+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:32.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:32.831+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:32.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:32.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:32.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:32.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:32.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:32.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:32.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:32.713+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:32.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:32.653+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:32.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:32.653+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:32.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:32.653+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:32.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:32.653+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:33.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.579+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:18:33.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.579+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:18:33.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.579+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8010f70 tx=0x7fcff002a9f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:18:33.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.579+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8010f70 tx=0x7fcff002a9f0 comp rx=0 tx=0).stop 2024-04-27T04:18:33.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.579+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:18:33.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.579+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:33.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.579+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:33.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.579+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:33.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.579+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4053090 con 0x7fcfe40447b0 2024-04-27T04:18:33.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.579+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40531d0 con 0x7fcff8101eb0 2024-04-27T04:18:33.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.579+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:18:33.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.580+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:18:33.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.580+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:33.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.580+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4053310 con 0x7fcfe40447b0 2024-04-27T04:18:33.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.581+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8008000 tx=0x7fcfe802f620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:18:33.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.582+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800cea0 con 0x7fcfe40447b0 2024-04-27T04:18:33.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.582+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe40447b0 2024-04-27T04:18:33.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:33.583+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe8013a90 con 0x7fcfe40447b0 2024-04-27T04:18:33.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:18:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:18:33.580+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 7917 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:18:33.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:33.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:33.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:33.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:33.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:33.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:33.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:33.807+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:33.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:33.617+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:33.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:33.617+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:33.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:33.617+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:33.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:33.617+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:33.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:33.687+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:33.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:33.687+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:33.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:33.687+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:33.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:33.688+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:34.843 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:34.585+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:34.843 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:34.585+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:34.843 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:34.585+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:34.844 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:34.585+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:34.844 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:34.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:34.844 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:34.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:34.844 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:34.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:34.844 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:34.654+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:35.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:35.030+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:18:35.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:35.030+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:18:35.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:34.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:35.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:34.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:35.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:34.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:35.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:34.845+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:35.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:35.797+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:35.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:35.797+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:35.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:35.797+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:35.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:35.797+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:35.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:35.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:35.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:35.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:35.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:35.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:35.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:35.704+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:35.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:35.599+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:35.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:35.599+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:35.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:35.599+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:35.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:35.600+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:36.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:36.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:36.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:36.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:36.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:36.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:36.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:36.736+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:36.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:36.589+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:36.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:36.589+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:36.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:36.589+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:36.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:36.589+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:36.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:36.758+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:36.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:36.758+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:36.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:36.758+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:36.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:36.758+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:37.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:37.771+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:37.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:37.771+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:37.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:37.771+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:37.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:37.773+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:37.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:37.689+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:37.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:37.689+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:37.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:37.689+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:37.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:37.690+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:37.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:37.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:37.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:37.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:37.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:37.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:37.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:37.562+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:38.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.580+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:18:38.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.580+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:18:38.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.580+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8008000 tx=0x7fcfe802f620 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:18:38.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.580+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8008000 tx=0x7fcfe802f620 comp rx=0 tx=0).stop 2024-04-27T04:18:38.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.580+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:18:38.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.580+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:38.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.580+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:38.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.580+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:38.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.580+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4053090 con 0x7fcfe4054300 2024-04-27T04:18:38.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.580+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40531d0 con 0x7fcff8101eb0 2024-04-27T04:18:38.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.581+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:18:38.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.581+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:18:38.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.581+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:38.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.581+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4053310 con 0x7fcfe4054300 2024-04-27T04:18:38.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.581+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fcff0034010 tx=0x7fcfe8008590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:18:38.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.582+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe4054300 2024-04-27T04:18:38.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.582+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fcfe4054300 2024-04-27T04:18:38.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:38.583+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcff0059040 con 0x7fcfe4054300 2024-04-27T04:18:38.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:18:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:18:38.581+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 7980 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:18:38.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:38.587+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:38.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:38.587+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:38.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:38.587+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:38.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:38.588+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:38.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:38.814+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:38.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:38.814+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:38.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:38.814+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:38.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:38.816+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:38.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:38.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:38.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:38.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:38.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:38.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:38.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:38.654+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:39.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:39.539+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:39.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:39.539+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:39.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:39.539+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:39.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:39.540+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:39.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:39.770+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:39.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:39.770+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:39.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:39.770+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:39.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:39.771+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:39.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:39.677+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:39.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:39.677+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:39.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:39.677+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:39.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:39.678+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:40.803 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:40.540+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:40.803 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:40.540+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:40.803 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:40.540+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:40.804 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:40.541+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:40.804 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:40.662+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:40.804 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:40.662+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:40.804 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:40.662+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:40.804 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:40.663+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:41.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:40.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:41.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:40.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:41.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:40.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:41.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:40.806+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:41.830 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:41.545+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:41.831 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:41.545+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:41.831 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:41.545+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:41.831 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:41.546+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:41.831 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:41.659+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:41.831 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:41.659+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:41.831 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:41.659+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:41.831 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:41.660+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:42.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:41.832+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:42.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:41.832+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:42.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:41.832+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:42.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:41.833+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:42.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:42.650+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:42.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:42.650+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:42.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:42.650+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:42.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:42.651+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:42.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:42.795+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:42.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:42.795+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:42.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:42.795+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:42.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:42.796+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:42.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:42.595+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:42.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:42.595+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:42.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:42.595+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:42.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:42.596+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:43.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.581+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-27T04:18:43.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.581+0000 7fcff6d9d700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:18:43.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.581+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fcff0034010 tx=0x7fcfe8008590 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:18:43.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.581+0000 7fcff6d9d700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fcff0034010 tx=0x7fcfe8008590 comp rx=0 tx=0).stop 2024-04-27T04:18:43.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.581+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:18:43.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.581+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:43.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.581+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:43.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.581+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:43.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.581+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e6c0 con 0x7fcfe40447b0 2024-04-27T04:18:43.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.581+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4053090 con 0x7fcff8101eb0 2024-04-27T04:18:43.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.582+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:18:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.582+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:18:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.582+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.582+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40531d0 con 0x7fcfe40447b0 2024-04-27T04:18:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.583+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40102a0 tx=0x7fcff0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:18:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.583+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe800cea0 con 0x7fcfe40447b0 2024-04-27T04:18:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.583+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe40447b0 2024-04-27T04:18:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:43.584+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe801e780 con 0x7fcfe40447b0 2024-04-27T04:18:43.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:18:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:18:43.582+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8042 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:18:43.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:43.763+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:43.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:43.763+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:43.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:43.763+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:43.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:43.764+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:43.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:43.693+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:43.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:43.693+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:43.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:43.693+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:43.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:43.693+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:43.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:43.620+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:43.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:43.620+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:43.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:43.620+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:43.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:43.621+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:44.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:44.648+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:44.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:44.648+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:44.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:44.648+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:44.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:44.648+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:44.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:44.728+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:44.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:44.728+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:44.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:44.728+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:44.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:44.729+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:44.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:44.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:44.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:44.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:44.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:44.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:44.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:44.738+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:45.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:45.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:45.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:45.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:45.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:45.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:45.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:45.739+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:45.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:45.670+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:45.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:45.670+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:45.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:45.670+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:45.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:45.671+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:45.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:45.723+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:45.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:45.723+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:45.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:45.723+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:45.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:45.724+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:47.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:46.701+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:47.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:46.701+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:47.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:46.701+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:47.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:46.701+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:47.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:46.696+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:47.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:46.696+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:47.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:46.696+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:47.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:46.698+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:47.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:46.715+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:47.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:46.715+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:47.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:46.715+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:47.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:46.716+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:48.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:47.722+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:48.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:47.722+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:48.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:47.722+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:48.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:47.723+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:48.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:47.728+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:48.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:47.728+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:48.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:47.728+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:48.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:47.729+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:48.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:47.716+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:48.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:47.716+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:48.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:47.716+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:48.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:47.717+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:48.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.581+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:18:48.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.581+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:18:48.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.582+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40102a0 tx=0x7fcff0000bc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:18:48.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.582+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40102a0 tx=0x7fcff0000bc0 comp rx=0 tx=0).stop 2024-04-27T04:18:48.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.582+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 msgr2=0x7fcfe4044b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:18:48.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.582+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:48.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.582+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:48.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.582+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:48.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.582+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe404e6c0 con 0x7fcfe4054300 2024-04-27T04:18:48.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.582+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4053090 con 0x7fcff8101eb0 2024-04-27T04:18:48.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.582+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:18:48.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.583+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:18:48.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.583+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:48.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.583+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe40531d0 con 0x7fcfe4054300 2024-04-27T04:18:48.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.583+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009680 tx=0x7fcfe801eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:18:48.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.585+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe4054300 2024-04-27T04:18:48.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.585+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8037ee0 con 0x7fcfe4054300 2024-04-27T04:18:48.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:48.588+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe801e780 con 0x7fcfe4054300 2024-04-27T04:18:48.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:18:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:18:48.582+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8104 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:18:48.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:48.735+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:48.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:48.735+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:48.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:48.735+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:48.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:48.735+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:48.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:48.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:48.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:48.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:48.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:48.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:48.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:48.719+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:48.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:48.722+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:48.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:48.722+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:48.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:48.722+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:48.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:48.723+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:49.986 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:49.738+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:49.987 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:49.738+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:49.987 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:49.738+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:49.987 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:49.739+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:49.987 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:49.729+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:49.987 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:49.729+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:49.987 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:49.729+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:49.987 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:49.729+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:49.988 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:49.676+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:49.988 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:49.676+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:49.988 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:49.676+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:49.988 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:49.677+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:50.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:50.388+0000 7fcff5d9b700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 msgr2=0x7fcfe403bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6828/1981832083 2024-04-27T04:18:50.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:50.388+0000 7fcff5d9b700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:6828/1981832083,v1:172.21.15.73:6829/1981832083] conn(0x7fcfe40398c0 0x7fcfe403bd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-27T04:18:50.956 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:50.705+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:50.956 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:50.705+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:50.956 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:50.706+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:50.956 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:50.706+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:50.956 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:50.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:50.957 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:50.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:50.957 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:50.722+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:50.957 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:50.722+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:50.957 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:50.706+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:50.957 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:50.706+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:50.957 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:50.706+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:50.957 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:50.707+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:51.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:51.705+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:51.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:51.705+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:51.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:51.705+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:51.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:51.705+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:51.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:51.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:51.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:51.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:51.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:51.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:51.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:51.693+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:51.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:51.694+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:51.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:51.694+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:51.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:51.694+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:51.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:51.695+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:53.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:52.683+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:53.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:52.683+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:53.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:52.683+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:53.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:52.684+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:53.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:52.719+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:53.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:52.719+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:53.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:52.719+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:53.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:52.719+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:53.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:52.747+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:53.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:52.747+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:53.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:52.747+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:53.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:52.748+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.582+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-27T04:18:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.583+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T04:18:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.583+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009680 tx=0x7fcfe801eca0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T04:18:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.583+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009680 tx=0x7fcfe801eca0 comp rx=0 tx=0).stop 2024-04-27T04:18:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.583+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 msgr2=0x7fcfe4052450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T04:18:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.583+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe4054300 0x7fcfe4052450 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.583+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.583+0000 7fcfef7fe700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T04:18:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.583+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4053090 con 0x7fcfe40447b0 2024-04-27T04:18:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.583+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40531d0 con 0x7fcff8101eb0 2024-04-27T04:18:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.583+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T04:18:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.583+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 msgr2=0x7fcfe4051f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T04:18:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.583+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcff8101eb0 0x7fcfe4051f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T04:18:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.584+0000 7fcff659c700 1 -- 172.21.15.46:0/1077475855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fcfe4053310 con 0x7fcfe40447b0 2024-04-27T04:18:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.584+0000 7fcff659c700 1 --2- 172.21.15.46:0/1077475855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe40447b0 0x7fcfe4044b90 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8019ff0 tx=0x7fcfe803a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T04:18:53.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.585+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8043d30 con 0x7fcfe40447b0 2024-04-27T04:18:53.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.585+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe804f670 con 0x7fcfe40447b0 2024-04-27T04:18:53.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:18:53.585+0000 7fcfef7fe700 1 -- 172.21.15.46:0/1077475855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 23) v1 ==== 46705+0+0 (secure 0 0 0) 0x7fcfe801e780 con 0x7fcfe40447b0 2024-04-27T04:18:53.728 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:18:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:18:53.583+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8167 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:18:53.728 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:53.676+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:53.728 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:53.676+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:53.728 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:53.676+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:53.728 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:53.677+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:53.728 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:53.659+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:53.729 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:53.659+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:53.729 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:53.659+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:53.729 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:53.659+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:54.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:53.728+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:54.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:53.728+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:54.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:53.728+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:54.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:53.729+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:54.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:54.646+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:54.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:54.646+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:54.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:54.646+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:54.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:54.647+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:54.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:54.676+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:54.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:54.676+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:54.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:54.676+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:54.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:54.677+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:54.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:54.711+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:54.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:54.711+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:54.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:54.711+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:54.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:54.711+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:55.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:55.688+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:55.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:55.688+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:55.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:55.688+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:55.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:55.689+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:55.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:55.637+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:55.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:55.637+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:55.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:55.637+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:55.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:55.637+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:55.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:55.681+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:55.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:55.681+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:55.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:55.681+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:55.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:55.681+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:56.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:56.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:56.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:56.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:56.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:56.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:56.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:56.668+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:56.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:56.661+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:56.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:56.661+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:56.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:56.661+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:56.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:56.662+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:56.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:56.680+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:56.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:56.680+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:56.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:56.680+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:56.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:56.680+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:57.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:57.669+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:57.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:57.669+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:57.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:57.669+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:57.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:57.669+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:57.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:57.647+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:57.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:57.647+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:57.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:57.647+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:57.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:57.647+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:57.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:57.662+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:57.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:57.662+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:57.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:57.662+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:57.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:57.663+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:57.970 INFO:teuthology.orchestra.run.smithi046.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-04-27T04:18:58.709 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:18:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:18:58.584+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8231 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:18:58.709 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:58.624+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:58.710 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:58.624+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:58.710 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:58.624+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:58.710 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:58.625+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:58.710 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:58.645+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:58.710 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:58.645+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:58.710 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:58.645+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:58.710 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:58.646+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:58.711 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:58.671+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:58.711 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:58.671+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:58.711 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:58.671+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:58.711 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:58.672+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:58.795 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -e sha1=4e213a58e787437ff508e6c890ccd5fd32dc96b5 -- bash -c 'ceph orch ps' 2024-04-27T04:18:59.016 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/mon.smithi046/config 2024-04-27T04:18:59.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:59.701+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:59.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:59.701+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:59.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:59.701+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:18:59.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:18:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:18:59.702+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:18:59.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:59.693+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:59.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:59.693+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:59.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:59.693+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:18:59.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:18:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:18:59.694+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:18:59.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:59.656+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:59.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:59.656+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:59.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:59.656+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:18:59.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:18:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:18:59.657+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:00.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:00.677+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:00.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:00.677+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:00.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:00.677+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:00.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:00.677+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:00.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:00.659+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:00.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:00.659+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:00.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:00.659+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:00.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:00.659+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:00.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:00.676+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:00.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:00.676+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:00.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:00.676+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:00.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:00.676+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:01.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:01.635+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:01.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:01.635+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:01.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:01.635+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:01.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:01.636+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:01.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:01.657+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:01.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:01.657+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:01.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:01.657+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:01.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:01.658+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:01.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:01.649+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:01.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:01.649+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:01.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:01.649+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:01.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:01.650+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:02.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:02.645+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:02.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:02.645+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:02.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:02.645+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:02.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:02.647+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:02.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:02.617+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:02.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:02.617+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:02.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:02.617+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:02.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:02.618+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:02.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:02.633+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:02.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:02.633+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:02.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:02.633+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:02.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:02.634+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:03.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:19:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:19:03.586+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8293 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:19:03.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:03.675+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:03.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:03.675+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:03.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:03.675+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:03.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:03.676+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:03.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:03.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:03.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:03.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:03.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:03.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:03.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:03.669+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:03.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:03.640+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:03.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:03.640+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:03.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:03.640+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:03.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:03.641+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:04.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:04.616+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:04.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:04.616+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:04.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:04.616+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:04.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:04.617+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:04.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:04.710+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:04.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:04.710+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:04.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:04.710+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:04.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:04.711+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:04.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:04.670+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:04.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:04.670+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:04.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:04.670+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:04.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:04.671+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:05.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:05.709+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:05.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:05.709+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:05.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:05.709+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:05.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:05.711+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:05.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:05.667+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:05.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:05.667+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:05.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:05.667+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:05.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:05.668+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:05.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:05.745+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:05.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:05.745+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:05.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:05.745+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:05.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:05.746+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:06.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:06.686+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:06.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:06.686+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:06.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:06.686+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:06.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:06.687+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:06.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:06.665+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:06.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:06.665+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:06.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:06.665+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:06.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:06.665+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:06.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:06.744+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:06.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:06.744+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:06.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:06.744+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:06.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:06.744+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:08.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:07.687+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:08.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:07.687+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:08.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:07.687+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:08.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:07.689+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:08.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:07.746+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:08.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:07.746+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:08.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:07.746+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:08.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:07.746+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:08.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:07.734+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:08.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:07.734+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:08.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:07.734+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:08.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:07.735+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:08.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:19:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:19:08.587+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8356 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:19:08.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:08.729+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:08.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:08.729+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:08.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:08.729+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:08.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:08.729+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:08.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:08.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:08.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:08.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:08.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:08.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:08.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:08.717+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:08.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:08.784+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:08.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:08.784+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:08.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:08.784+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:08.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:08.785+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:09.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:09.685+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:09.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:09.685+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:09.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:09.685+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:09.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:09.686+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:09.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:09.826+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:09.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:09.826+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:09.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:09.826+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:09.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:09.826+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:09.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:09.671+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:09.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:09.671+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:09.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:09.671+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:09.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:09.671+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:10.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:10.817+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:10.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:10.817+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:10.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:10.817+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:10.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:10.817+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:10.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:10.641+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:10.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:10.641+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:10.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:10.641+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:10.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:10.641+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:10.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:10.679+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:10.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:10.679+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:10.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:10.679+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:10.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:10.681+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:11.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:11.809+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:11.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:11.809+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:11.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:11.809+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:11.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:11.810+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:11.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:11.597+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:11.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:11.597+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:11.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:11.597+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:11.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:11.598+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:11.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:11.677+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:11.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:11.677+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:11.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:11.677+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:11.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:11.678+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:12.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:12.658+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:12.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:12.658+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:12.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:12.658+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:12.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:12.659+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:12.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:12.793+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:12.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:12.793+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:12.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:12.793+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:12.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:12.794+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:12.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:12.607+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:12.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:12.607+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:12.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:12.607+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:12.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:12.608+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:13.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:19:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:19:13.588+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8419 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:19:13.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:13.612+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:13.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:13.612+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:13.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:13.612+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:13.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:13.613+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:13.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:13.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:13.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:13.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:13.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:13.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:13.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:13.649+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:13.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:13.782+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:13.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:13.782+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:13.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:13.782+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:13.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:13.783+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:14.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:14.603+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:14.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:14.603+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:14.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:14.603+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:14.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:14.604+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:14.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:14.624+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:14.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:14.624+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:14.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:14.624+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:14.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:14.625+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:14.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:14.741+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:14.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:14.741+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:14.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:14.741+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:14.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:14.742+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:15.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:15.791+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:15.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:15.791+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:15.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:15.791+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:15.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:15.792+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:15.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:15.567+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:15.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:15.567+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:15.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:15.567+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:15.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:15.568+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:15.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:15.625+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:15.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:15.625+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:15.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:15.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:15.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:15.626+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:16.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:16.781+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:16.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:16.781+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:16.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:16.781+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:16.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:16.781+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:16.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:16.587+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:16.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:16.587+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:16.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:16.587+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:16.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:16.588+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:16.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:16.553+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:16.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:16.553+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:16.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:16.553+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:16.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:16.554+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:17.920 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:17.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:17.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:17.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:17.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:17.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:17.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:17.770+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:17.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:17.596+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:17.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:17.596+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:17.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:17.596+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:17.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:17.597+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:17.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:17.578+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:17.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:17.578+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:17.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:17.578+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:17.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:17.579+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:18.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:19:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:19:18.589+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8482 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:19:18.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:18.558+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:18.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:18.558+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:18.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:18.558+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:18.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:18.560+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:18.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:18.598+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:18.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:18.598+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:18.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:18.598+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:18.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:18.599+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:18.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:18.740+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:18.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:18.740+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:18.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:18.740+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:18.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:18.740+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:19.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:19.700+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:19.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:19.700+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:19.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:19.700+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:19.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:19.701+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:19.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:19.634+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:19.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:19.634+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:19.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:19.634+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:19.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:19.636+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:19.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:19.538+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:19.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:19.538+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:19.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:19.538+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:19.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:19.538+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:20.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:20.713+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:20.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:20.713+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:20.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:20.713+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:20.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:20.714+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:20.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:20.594+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:20.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:20.594+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:20.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:20.594+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:20.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:20.595+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:20.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:20.510+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:20.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:20.510+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:20.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:20.510+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:20.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:20.512+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:21.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:21.568+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:21.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:21.568+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:21.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:21.568+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:21.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:21.569+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:21.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:21.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:21.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:21.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:21.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:21.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:21.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:21.493+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:21.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:21.711+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:21.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:21.711+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:21.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:21.711+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:21.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:21.711+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:22.700 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:22.570+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:22.700 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:22.570+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:22.700 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:22.570+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:22.701 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:22.571+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:22.701 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:22.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:22.701 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:22.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:22.701 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:22.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:22.701 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:22.492+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:23.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:22.702+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:23.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:22.702+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:23.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:22.702+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:23.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:22.702+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:23.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:19:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:19:23.590+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8545 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:19:23.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:23.490+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:23.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:23.490+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:23.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:23.490+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:23.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:23.491+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:23.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:23.549+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:23.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:23.549+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:23.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:23.549+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:23.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:23.550+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:23.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:23.698+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:23.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:23.698+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:23.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:23.698+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:23.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:23.699+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:24.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:24.510+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:24.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:24.510+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:24.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:24.510+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:24.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:24.511+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:24.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:24.543+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:24.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:24.543+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:24.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:24.543+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:24.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:24.543+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:24.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:24.710+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:24.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:24.710+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:24.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:24.710+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:24.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:24.711+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:25.499+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:25.499+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:25.499+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:25.500+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:25.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:25.503+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:25.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:25.503+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:25.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:25.503+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:25.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:25.504+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:25.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:25.692+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:25.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:25.692+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:25.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:25.692+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:25.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:25.693+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:26.490 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:26.459+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:26.490 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:26.459+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:26.490 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:26.459+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:26.490 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:26.460+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:26.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:26.491+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:26.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:26.491+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:26.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:26.491+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:26.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:26.492+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:26.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:26.734+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:26.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:26.734+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:26.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:26.734+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:26.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:26.734+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:27.716 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:27.481+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:27.716 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:27.481+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:27.716 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:27.481+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:27.716 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:27.482+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:27.716 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:27.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:27.716 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:27.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:27.716 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:27.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:27.716 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:27.461+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:28.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:27.717+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:28.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:27.717+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:28.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:27.717+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:28.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:27.718+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:28.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:19:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:19:28.591+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8610 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:19:28.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:28.731+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:28.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:28.731+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:28.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:28.732+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:28.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:28.733+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:28.490+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:28.490+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:28.490+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:28.491+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:28.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:28.499+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:28.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:28.499+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:28.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:28.499+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:28.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:28.500+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:29.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:29.459+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:29.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:29.459+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:29.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:29.459+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:29.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:29.459+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:29.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:29.508+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:29.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:29.508+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:29.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:29.508+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:29.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:29.509+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:29.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:29.695+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:29.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:29.695+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:29.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:29.695+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:29.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:29.697+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:30.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:30.417+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:30.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:30.417+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:30.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:30.417+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:30.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:30.418+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:30.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:30.504+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:30.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:30.504+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:30.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:30.504+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:30.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:30.505+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:31.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:30.739+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:31.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:30.739+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:31.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:30.739+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:31.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:30.741+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:31.546 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:31.392+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:31.546 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:31.392+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:31.546 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:31.392+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:31.547 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:31.393+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:31.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:31.548+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:31.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:31.548+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:31.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:31.548+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:31.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:31.548+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:31.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:31.706+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:31.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:31.706+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:31.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:31.706+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:31.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:31.707+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:32.658 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:32.380+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:32.658 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:32.380+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:32.658 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:32.380+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:32.658 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:32.381+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:32.658 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:32.547+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:32.658 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:32.547+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:32.658 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:32.547+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:32.658 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:32.547+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:32.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:32.659+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:32.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:32.659+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:32.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:32.659+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:32.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:32.660+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:33.552 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:33.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:33.552 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:33.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:33.552 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:33.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:33.552 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:33.378+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:33.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:19:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:19:33.592+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8673 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:19:33.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:33.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:33.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:33.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:33.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:33.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:33.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:33.555+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:33.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:33.622+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:33.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:33.622+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:33.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:33.622+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:33.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:33.623+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:34.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:34.508+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:34.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:34.508+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:34.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:34.508+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:34.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:34.509+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:34.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:34.398+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:34.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:34.398+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:34.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:34.398+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:34.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:34.399+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:34.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:34.628+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:34.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:34.628+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:34.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:34.628+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:34.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:34.629+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:35.651 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:35.371+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:35.651 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:35.371+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:35.651 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:35.371+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:35.651 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:35.372+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:35.651 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:35.545+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:35.651 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:35.545+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:35.651 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:35.545+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:35.651 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:35.547+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:35.920 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:35.652+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:35.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:35.652+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:35.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:35.652+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:35.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:35.653+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:36.573 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:36.386+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:36.573 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:36.386+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:36.574 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:36.386+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:36.574 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:36.388+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:36.920 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:36.669+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:36.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:36.669+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:36.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:36.669+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:36.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:36.670+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:36.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:36.575+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:36.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:36.575+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:36.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:36.575+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:36.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:36.576+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:37.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:37.647+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:37.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:37.647+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:37.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:37.647+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:37.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:37.648+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:37.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:37.582+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:37.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:37.582+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:37.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:37.582+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:37.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:37.583+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:37.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:37.401+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:37.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:37.401+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:37.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:37.401+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:37.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:37.401+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:38.662 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:19:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:19:38.593+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8737 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:19:38.662 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:38.361+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:38.662 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:38.361+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:38.662 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:38.361+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:38.662 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:38.361+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:38.663 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:38.605+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:38.663 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:38.605+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:38.663 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:38.605+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:38.663 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:38.605+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:38.920 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:38.663+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:38.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:38.663+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:38.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:38.663+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:38.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:38.664+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:39.648 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:39.328+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:39.648 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:39.328+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:39.648 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:39.328+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:39.648 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:39.328+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:39.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:39.693+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:39.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:39.693+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:39.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:39.693+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:39.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:39.694+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:39.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:39.649+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:39.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:39.649+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:39.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:39.649+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:39.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:39.650+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:40.628 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:40.370+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:40.628 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:40.370+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:40.628 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:40.370+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:40.629 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:40.371+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:40.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:40.630+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:40.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:40.630+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:40.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:40.630+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:40.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:40.631+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:40.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:40.718+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:40.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:40.718+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:40.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:40.718+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:40.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:40.719+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:41.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:41.341+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:41.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:41.341+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:41.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:41.341+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:41.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:41.342+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:42.101 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:41.676+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:42.102 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:41.676+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:42.102 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:41.676+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:42.102 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:41.677+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:42.102 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:41.730+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:42.102 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:41.730+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:42.102 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:41.730+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:42.102 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:41.731+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:42.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:42.306+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:42.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:42.306+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:42.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:42.306+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:42.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:42.307+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:42.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:42.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:42.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:42.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:42.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:42.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:42.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:42.653+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:42.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:42.742+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:42.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:42.742+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:42.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:42.742+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:42.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:42.743+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:43.593 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:43.327+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:43.593 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:43.327+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:43.593 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:43.327+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:43.593 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:43.329+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:43.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:19:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:19:43.594+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8799 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:19:43.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:43.637+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:43.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:43.637+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:43.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:43.637+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:43.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:43.638+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:43.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:43.741+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:43.921 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:43.741+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:43.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:43.741+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:43.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:43.742+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:44.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:44.362+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:44.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:44.362+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:44.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:44.362+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:44.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:44.363+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:45.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:44.717+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:45.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:44.717+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:45.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:44.717+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:45.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:44.718+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:45.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:44.684+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:45.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:44.684+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:45.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:44.684+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:45.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:44.685+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:45.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:45.351+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:45.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:45.351+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:45.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:45.351+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:45.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:45.351+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:46.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:45.768+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:46.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:45.768+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:46.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:45.768+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:46.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:45.768+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:46.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:45.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:46.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:45.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:46.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:45.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:46.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:45.732+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:46.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:46.400+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:46.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:46.400+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:46.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:46.400+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:46.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:46.401+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:47.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:46.730+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:47.189 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:46.730+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:47.189 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:46.730+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:47.189 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:46.731+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:47.190 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:46.796+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:47.190 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:46.796+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:47.190 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:46.796+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:47.190 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:46.797+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:47.701 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:47.444+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:47.701 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:47.444+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:47.701 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:47.444+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:47.701 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:47.445+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:48.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:47.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:48.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:47.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:48.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:47.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:48.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:47.703+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:48.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:47.843+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:48.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:47.843+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:48.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:47.843+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:48.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:47.843+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:48.756 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:19:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:19:48.595+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8861 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:19:48.757 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:48.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:48.757 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:48.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:48.757 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:48.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:48.757 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:48.461+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:48.757 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:48.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:48.757 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:48.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:48.757 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:48.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:48.757 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:48.705+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:49.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:48.869+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:49.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:48.869+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:49.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:48.869+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:49.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:48.870+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:49.721 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:49.452+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:49.721 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:49.452+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:49.721 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:49.452+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:49.721 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:49.453+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:50.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:49.722+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:50.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:49.722+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:50.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:49.722+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:50.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:49.723+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:50.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:49.866+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:50.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:49.866+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:50.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:49.866+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:50.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:49.867+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:50.906 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:50.488+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:50.906 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:50.488+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:50.906 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:50.488+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:50.906 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:50.489+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:50.906 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:50.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:50.907 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:50.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:50.907 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:50.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:50.907 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:50.714+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:51.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:50.907+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:51.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:50.907+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:51.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:50.907+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:51.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:50.908+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:51.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:51.474+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:51.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:51.474+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:51.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:51.475+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:51.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:51.476+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:52.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:51.713+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:52.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:51.713+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:52.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:51.713+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:52.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:51.714+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:52.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:51.898+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:52.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:51.898+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:52.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:51.898+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:52.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:51.898+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:52.708 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:52.441+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:52.709 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:52.441+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:52.709 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:52.441+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:52.709 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:52.442+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:53.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:52.710+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:53.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:52.710+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:53.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:52.710+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:53.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:52.711+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:53.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:52.926+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:53.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:52.926+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:53.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:52.926+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:53.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:52.927+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:53.876 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:19:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:19:53.596+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8923 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:19:53.876 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:53.458+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:53.877 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:53.458+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:53.877 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:53.458+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:53.877 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:53.458+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:53.877 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:53.687+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:53.877 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:53.687+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:53.877 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:53.687+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:53.877 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:53.688+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:54.388 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:53.959+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:54.388 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:53.959+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:54.388 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:53.959+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:54.388 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:53.960+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:54.715 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:54.450+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:54.715 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:54.450+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:54.715 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:54.450+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:54.715 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:54.451+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:54.971 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:54.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:54.971 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:54.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:54.971 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:54.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:54.972 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:54.717+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:55.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:54.973+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:55.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:54.973+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:55.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:54.973+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:55.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:54.974+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:55.748 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:55.447+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:55.748 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:55.447+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:55.748 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:55.447+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:55.748 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:55.448+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:56.002 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:55.749+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:56.002 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:55.749+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:56.002 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:55.749+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:56.003 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:55.750+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:56.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:56.004+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:56.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:56.004+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:56.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:56.004+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:56.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:56.004+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:56.791 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:56.473+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:56.791 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:56.473+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:56.791 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:56.473+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:56.791 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:56.475+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:57.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:56.977+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:57.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:56.977+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:57.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:56.977+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:57.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:56.978+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:57.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:56.792+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:57.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:56.792+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:57.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:56.792+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:57.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:56.794+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:57.758 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:57.479+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:57.758 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:57.479+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:57.759 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:57.479+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:57.759 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:57.481+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:58.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:57.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:58.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:57.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:58.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:57.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:58.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:57.761+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:58.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:57.955+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:58.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:57.955+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:58.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:57.955+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:58.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:57.955+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:58.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:19:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:19:58.597+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 8986 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:19:58.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:58.486+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:58.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:58.486+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:58.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:58.486+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:58.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:58.488+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:58.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:58.725+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:58.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:58.725+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:58.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:58.725+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:19:58.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:58.726+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:19:59.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:58.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:59.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:58.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:59.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:58.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:19:59.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:58.997+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:19:59.764 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:59.446+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:59.765 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:59.446+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:59.765 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:59.446+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:19:59.765 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:19:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:19:59.447+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:00.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:59.766+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:00.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:59.766+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:00.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:59.766+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:00.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:19:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:19:59.768+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:00.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:59.983+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:00.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:59.983+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:00.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:59.983+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:00.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:19:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:19:59.984+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:00.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:00.404+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:00.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:00.404+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:00.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:00.404+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:00.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:00.404+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:01.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:00.946+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:01.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:00.946+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:01.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:00.946+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:01.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:00.947+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:01.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:00.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:01.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:00.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:01.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:00.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:01.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:00.758+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:01.498 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:01.380+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:01.498 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:01.380+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:01.498 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:01.380+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:01.498 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:01.381+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:02.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:01.761+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:02.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:01.761+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:02.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:01.761+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:02.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:01.762+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:02.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:01.995+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:02.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:01.995+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:02.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:01.995+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:02.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:01.995+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:02.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:02.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:02.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:02.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:02.376+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:03.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:02.771+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:03.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:02.771+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:03.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:02.771+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:03.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:02.772+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:03.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:02.987+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:03.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:02.987+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:03.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:02.987+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:03.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:02.987+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:03.798 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:20:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:20:03.598+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9047 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:20:03.798 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:03.422+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:03.798 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:03.422+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:03.798 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:03.422+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:03.798 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:03.423+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:04.116 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:03.987+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:04.116 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:03.987+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:04.116 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:03.987+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:04.116 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:03.988+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:04.116 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:03.799+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:04.116 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:03.799+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:04.116 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:03.799+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:04.116 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:03.800+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:04.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:04.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:04.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:04.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:04.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:04.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:04.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:04.378+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:05.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:04.846+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:05.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:04.846+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:05.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:04.846+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:05.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:04.847+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:05.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:04.956+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:05.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:04.956+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:05.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:04.956+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:05.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:04.956+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:05.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:05.369+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:05.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:05.369+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:05.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:05.369+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:05.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:05.370+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:05.896+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:05.896+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:05.896+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:05.897+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:06.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:05.946+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:06.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:05.946+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:05.946+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:05.947+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:06.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:06.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:06.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:06.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:06.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:06.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:06.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:06.375+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:07.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:06.847+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:07.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:06.847+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:07.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:06.847+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:07.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:06.848+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:07.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:06.901+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:07.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:06.901+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:07.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:06.901+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:07.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:06.902+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:07.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:07.398+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:07.703 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:07.398+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:07.704 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:07.398+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:07.704 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:07.399+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:08.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:07.859+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:08.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:07.859+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:08.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:07.859+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:08.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:07.860+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:08.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:07.914+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:08.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:07.914+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:08.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:07.914+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:08.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:07.914+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:08.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:20:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:20:08.599+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9110 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:20:08.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:08.401+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:08.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:08.401+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:08.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:08.401+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:08.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:08.402+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:09.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:08.877+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:09.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:08.877+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:09.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:08.877+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:09.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:08.879+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:09.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:08.836+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:09.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:08.836+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:09.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:08.836+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:09.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:08.836+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:09.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:09.415+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:09.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:09.415+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:09.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:09.415+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:09.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:09.416+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:09.885+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:09.886+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:09.886+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:09.886+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:10.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:09.876+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:10.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:09.876+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:10.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:09.876+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:10.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:09.877+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:10.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:10.438+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:10.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:10.438+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:10.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:10.438+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:10.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:10.439+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:11.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:10.930+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:11.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:10.930+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:11.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:10.930+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:11.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:10.930+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:11.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:10.829+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:11.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:10.829+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:11.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:10.829+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:11.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:10.830+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:11.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:11.416+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:11.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:11.416+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:11.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:11.416+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:11.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:11.417+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:12.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:11.932+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:12.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:11.932+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:12.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:11.932+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:12.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:11.933+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:11.875+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:11.875+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:11.875+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:11.876+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:12.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:12.408+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:12.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:12.408+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:12.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:12.408+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:12.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:12.409+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:13.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:12.945+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:13.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:12.945+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:13.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:12.945+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:13.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:12.946+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:13.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:12.841+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:13.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:12.841+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:13.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:12.841+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:13.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:12.842+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:13.599 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:13.375+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:13.599 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:13.375+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:13.599 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:13.375+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:13.599 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:13.377+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:13.889 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:20:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:20:13.601+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9172 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:20:14.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:13.891+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:14.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:13.891+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:14.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:13.891+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:14.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:13.891+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:14.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:13.992+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:14.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:13.992+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:14.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:13.992+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:14.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:13.993+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:14.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:14.395+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:14.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:14.395+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:14.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:14.395+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:14.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:14.396+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:15.414 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:14.932+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:15.415 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:14.932+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:15.415 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:14.932+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:15.415 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:14.933+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:15.415 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:14.993+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:15.415 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:14.993+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:15.415 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:14.993+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:15.415 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:14.995+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:15.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:15.416+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:15.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:15.416+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:15.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:15.416+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:15.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:15.417+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:16.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:15.978+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:16.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:15.978+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:16.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:15.978+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:16.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:15.979+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:16.376 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:15.982+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:16.377 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:15.982+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:16.377 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:15.982+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:16.377 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:15.983+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:16.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:16.378+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:16.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:16.378+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:16.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:16.378+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:16.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:16.378+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:17.372 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:16.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:17.373 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:16.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:17.373 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:16.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:17.373 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:16.998+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:17.373 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:17.024+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:17.373 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:17.024+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:17.373 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:17.024+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:17.373 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:17.024+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:17.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:17.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:17.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:17.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:17.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:17.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:17.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:17.375+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:18.376 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:18.062+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:18.376 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:18.062+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:18.376 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:18.062+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:18.376 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:18.063+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:18.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:18.040+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:18.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:18.040+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:18.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:18.040+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:18.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:18.041+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:18.670 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:20:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:20:18.602+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9235 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:20:18.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:18.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:18.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:18.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:18.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:18.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:18.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:18.378+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:19.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:19.080+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:19.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:19.080+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:19.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:19.080+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:19.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:19.081+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:19.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:19.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:19.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:19.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:19.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:19.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:19.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:19.008+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:19.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:19.423+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:19.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:19.423+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:19.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:19.423+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:19.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:19.423+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:20.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:20.040+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:20.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:20.040+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:20.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:20.040+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:20.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:20.041+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:20.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:20.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:20.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:20.017+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:20.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:20.432+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:20.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:20.432+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:20.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:20.432+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:20.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:20.433+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:21.390 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:21.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:21.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:21.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:21.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:21.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:21.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:21.008+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:21.391 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:21.044+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:21.391 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:21.044+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:21.391 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:21.044+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:21.391 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:21.045+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:21.648 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:21.391+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:21.649 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:21.391+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:21.649 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:21.391+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:21.649 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:21.392+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:22.396 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:22.031+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:22.396 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:22.031+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:22.397 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:22.031+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:22.397 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:22.032+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:22.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:22.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:22.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:22.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:22.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:22.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:22.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:22.036+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:22.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:22.397+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:22.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:22.397+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:22.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:22.397+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:22.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:22.398+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:23.354 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:22.996+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:23.354 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:22.996+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:23.354 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:22.996+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:23.354 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:22.997+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:23.354 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:23.023+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:23.355 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:23.023+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:23.355 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:23.023+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:23.355 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:23.024+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:23.670 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:20:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:20:23.602+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9296 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:20:23.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:23.354+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:23.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:23.354+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:23.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:23.354+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:23.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:23.355+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:24.383 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:23.978+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:24.383 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:23.978+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:24.383 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:23.978+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:24.383 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:23.978+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:24.383 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:24.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:24.383 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:24.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:24.383 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:24.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:24.383 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:24.034+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:24.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:24.383+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:24.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:24.383+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:24.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:24.383+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:24.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:24.384+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:25.366 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:24.967+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:25.366 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:24.967+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:25.366 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:24.967+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:25.367 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:24.968+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:25.367 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:25.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:25.367 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:25.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:25.367 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:25.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:25.367 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:25.000+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:25.619 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:25.367+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:25.619 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:25.367+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:25.619 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:25.367+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:25.619 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:25.368+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:26.356 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:25.932+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:26.356 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:25.932+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:26.356 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:25.932+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:26.356 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:25.933+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:26.356 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:25.955+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:26.357 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:25.955+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:26.357 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:25.955+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:26.357 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:25.957+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:26.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:26.357+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:26.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:26.357+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:26.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:26.357+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:26.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:26.358+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:27.354 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:26.969+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:27.354 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:26.969+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:27.354 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:26.969+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:27.354 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:26.970+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:27.354 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:26.965+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:27.354 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:26.965+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:27.354 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:26.965+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:27.354 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:26.966+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:27.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:27.354+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:27.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:27.354+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:27.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:27.354+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:27.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:27.355+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:28.305 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:27.968+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:28.305 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:27.968+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:28.306 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:27.968+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:28.306 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:27.969+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:28.306 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:27.993+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:28.306 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:27.993+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:28.306 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:27.993+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:28.306 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:27.994+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:28.602 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:28.306+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:28.602 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:28.306+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:28.602 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:28.306+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:28.602 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:28.306+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:28.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:20:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:20:28.603+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9359 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:20:29.353 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:28.928+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:29.353 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:28.928+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:29.353 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:28.928+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:29.353 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:28.929+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:29.354 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:28.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:29.354 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:28.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:29.354 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:28.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:29.354 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:28.963+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:29.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:29.354+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:29.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:29.354+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:29.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:29.354+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:29.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:29.355+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:30.351 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:29.932+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:30.352 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:29.932+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:30.352 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:29.932+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:30.352 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:29.933+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:30.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:29.949+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:30.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:29.949+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:30.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:29.949+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:30.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:29.949+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:30.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:30.352+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:30.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:30.352+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:30.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:30.352+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:30.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:30.353+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:31.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:30.987+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:31.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:30.987+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:31.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:30.987+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:31.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:30.987+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:31.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:30.903+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:31.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:30.903+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:31.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:30.903+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:31.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:30.904+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:31.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:31.331+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:31.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:31.331+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:31.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:31.331+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:31.569 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:31.332+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:32.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:31.960+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:32.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:31.960+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:32.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:31.960+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:32.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:31.961+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:32.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:31.900+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:32.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:31.900+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:32.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:31.900+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:32.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:31.901+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:32.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:32.287+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:32.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:32.287+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:32.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:32.287+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:32.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:32.288+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:33.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:32.918+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:33.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:32.918+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:33.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:32.918+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:33.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:32.918+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:33.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:32.882+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:33.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:32.882+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:33.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:32.882+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:33.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:32.883+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:33.603 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:33.330+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:33.604 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:33.330+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:33.604 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:33.330+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:33.604 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:33.331+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:33.877 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:20:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:20:33.605+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9422 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:20:34.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:33.933+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:34.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:33.933+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:34.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:33.933+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:34.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:33.934+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:34.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:33.879+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:34.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:33.879+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:34.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:33.879+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:34.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:33.880+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:34.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:34.283+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:34.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:34.283+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:34.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:34.283+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:34.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:34.284+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:35.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:34.970+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:35.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:34.970+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:35.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:34.970+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:35.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:34.971+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:34.839+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:34.839+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:34.839+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:34.841+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:35.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:35.273+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:35.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:35.273+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:35.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:35.273+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:35.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:35.274+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:36.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:35.979+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:36.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:35.979+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:36.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:35.979+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:36.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:35.980+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:36.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:35.836+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:36.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:35.836+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:36.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:35.836+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:36.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:35.836+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:36.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:36.246+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:36.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:36.246+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:36.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:36.246+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:36.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:36.248+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:37.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:36.974+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:37.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:36.974+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:37.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:36.974+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:37.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:36.975+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:37.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:36.860+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:37.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:36.860+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:37.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:36.860+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:37.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:36.862+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:37.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:37.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:37.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:37.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:37.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:37.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:37.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:37.216+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:38.022 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:37.883+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:38.022 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:37.883+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:38.022 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:37.883+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:38.022 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:37.884+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:38.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:38.225+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:38.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:38.225+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:38.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:38.225+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:38.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:38.226+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:38.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:38.024+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:38.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:38.024+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:38.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:38.024+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:38.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:38.025+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:38.869 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:20:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:20:38.606+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9487 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:20:39.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:39.066+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:39.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:39.066+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:39.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:39.066+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:39.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:39.066+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:39.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:38.871+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:39.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:38.871+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:39.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:38.871+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:39.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:38.872+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:39.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:39.177+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:39.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:39.177+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:39.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:39.177+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:39.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:39.178+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:40.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:39.909+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:40.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:39.909+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:40.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:39.909+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:40.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:39.910+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:40.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:40.028+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:40.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:40.028+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:40.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:40.029+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:40.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:40.029+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:40.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:40.208+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:40.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:40.208+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:40.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:40.208+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:40.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:40.208+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:41.191 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:41.075+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:41.191 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:41.075+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:41.191 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:41.075+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:41.191 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:41.076+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:41.191 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:40.939+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:41.191 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:40.939+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:41.191 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:40.939+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:41.191 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:40.940+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:41.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:41.192+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:41.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:41.192+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:41.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:41.192+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:41.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:41.193+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:42.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:41.944+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:42.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:41.944+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:42.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:41.944+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:42.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:41.944+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:42.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:42.187+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:42.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:42.187+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:42.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:42.187+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:42.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:42.188+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:42.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:42.122+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:42.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:42.122+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:42.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:42.122+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:42.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:42.122+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:43.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:43.192+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:43.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:43.192+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:43.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:43.192+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:43.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:43.193+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:43.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:43.115+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:43.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:43.115+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:43.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:43.115+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:43.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:43.117+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:43.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:42.983+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:43.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:42.983+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:43.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:42.983+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:43.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:42.984+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:43.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:20:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:20:43.607+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9549 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:20:44.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:44.027+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:44.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:44.027+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:44.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:44.027+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:44.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:44.028+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:44.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:44.099+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:44.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:44.099+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:44.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:44.099+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:44.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:44.100+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:44.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:44.230+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:44.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:44.230+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:44.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:44.230+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:44.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:44.231+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:45.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:45.129+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:45.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:45.129+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:45.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:45.129+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:45.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:45.130+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:45.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:45.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:45.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:45.079+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:45.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:45.266+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:45.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:45.266+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:45.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:45.266+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:45.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:45.267+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:46.314 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:46.146+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:46.314 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:46.146+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:46.314 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:46.146+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:46.314 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:46.147+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:46.315 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:46.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:46.315 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:46.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:46.315 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:46.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:46.315 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:46.047+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:46.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:46.315+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:46.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:46.315+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:46.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:46.315+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:46.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:46.316+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:47.124 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:47.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:47.124 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:47.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:47.124 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:47.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:47.124 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:47.036+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:47.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:47.133+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:47.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:47.133+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:47.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:47.133+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:47.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:47.134+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:47.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:47.352+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:47.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:47.352+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:47.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:47.352+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:47.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:47.353+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:48.334 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:48.031+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:48.334 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:48.031+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:48.334 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:48.031+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:48.334 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:48.032+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:48.335 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:48.158+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:48.335 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:48.158+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:48.335 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:48.158+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:48.335 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:48.159+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:48.606 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:48.336+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:48.607 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:48.336+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:48.607 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:48.336+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:48.607 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:48.337+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:48.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:20:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:20:48.608+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9613 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:20:49.326 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:49.056+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:49.326 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:49.056+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:49.326 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:49.056+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:49.326 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:49.057+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:49.327 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:49.169+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:49.327 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:49.169+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:49.327 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:49.169+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:49.327 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:49.170+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:49.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:49.328+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:49.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:49.328+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:49.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:49.328+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:49.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:49.329+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:50.333 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:50.044+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:50.334 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:50.044+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:50.334 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:50.044+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:50.334 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:50.044+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:50.334 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:50.177+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:50.334 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:50.177+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:50.334 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:50.177+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:50.334 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:50.178+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:50.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:50.335+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:50.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:50.335+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:50.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:50.335+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:50.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:50.336+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:51.374 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:51.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:51.374 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:51.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:51.374 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:51.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:51.375 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:51.001+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:51.375 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:51.135+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:51.375 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:51.135+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:51.375 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:51.135+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:51.375 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:51.136+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:51.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:51.376+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:51.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:51.376+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:51.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:51.376+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:51.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:51.377+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:52.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:52.096+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:52.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:52.096+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:52.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:52.096+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:52.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:52.097+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:52.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:52.021+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:52.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:52.021+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:52.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:52.021+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:52.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:52.022+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:52.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:52.391+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:52.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:52.391+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:52.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:52.391+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:52.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:52.391+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:53.346 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:53.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:53.346 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:53.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:53.346 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:53.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:53.346 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:53.030+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:53.347 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:53.063+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:53.347 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:53.063+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:53.347 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:53.063+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:53.347 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:53.064+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:53.607 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:53.347+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:53.607 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:53.347+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:53.608 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:53.347+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:53.608 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:53.348+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:53.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:20:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:20:53.609+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9666 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:20:54.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:54.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:54.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:54.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:54.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:54.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:54.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:54.075+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:54.378 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:54.032+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:54.378 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:54.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:54.378 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:54.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:54.378 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:54.034+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:54.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:54.379+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:54.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:54.379+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:54.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:54.379+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:54.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:54.380+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:55.349 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:55.040+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:55.350 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:55.040+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:55.350 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:55.040+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:55.350 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:55.040+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:55.350 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:55.015+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:55.350 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:55.015+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:55.350 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:55.015+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:55.350 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:55.016+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:55.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:55.351+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:55.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:55.351+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:55.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:55.351+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:55.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:55.352+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:56.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:56.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:56.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:56.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:56.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:56.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:56.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:56.006+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:56.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:56.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:56.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:56.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:56.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:56.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:56.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:56.001+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:56.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:56.352+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:56.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:56.352+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:56.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:56.352+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:56.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:56.353+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:57.366 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:57.019+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:57.366 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:57.019+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:57.366 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:57.019+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:57.367 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:57.020+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:57.367 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:56.987+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:57.367 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:56.987+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:57.367 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:56.987+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:57.367 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:56.988+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:57.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:57.368+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:57.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:57.368+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:57.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:57.368+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:57.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:57.368+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:58.415 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:58.049+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:58.415 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:58.049+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:58.415 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:58.049+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:58.415 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:58.050+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:58.416 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:58.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:58.416 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:58.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:58.416 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:58.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:58.416 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:58.002+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:58.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:20:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:20:58.610+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9718 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:20:58.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:58.417+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:58.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:58.417+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:58.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:58.417+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:58.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:58.418+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:59.405 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:59.062+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:59.405 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:59.062+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:59.405 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:59.062+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:20:59.405 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:20:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:20:59.063+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:20:59.405 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:59.041+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:59.405 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:59.041+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:59.405 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:59.041+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:20:59.405 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:20:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:20:59.042+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:20:59.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:59.406+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:59.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:59.406+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:59.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:59.406+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:20:59.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:20:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:20:59.408+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:00.375 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:00.096+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:00.375 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:00.096+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:00.375 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:00.096+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:00.375 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:00.097+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:00.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:00.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:00.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:00.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:00.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:00.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:00.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:00.036+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:00.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:00.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:00.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:00.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:00.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:00.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:00.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:00.378+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:01.372 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:01.048+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:01.372 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:01.048+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:01.372 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:01.048+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:01.372 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:01.049+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:01.372 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:01.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:01.373 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:01.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:01.373 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:01.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:01.373 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:01.082+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:01.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:01.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:01.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:01.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:01.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:01.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:01.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:01.374+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:02.375 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:02.064+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:02.376 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:02.064+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:02.376 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:02.064+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:02.376 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:02.065+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:02.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:02.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:02.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:02.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:02.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:02.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:02.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:02.069+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:02.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:02.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:02.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:02.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:02.378+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:03.414 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:03.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:03.414 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:03.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:03.414 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:03.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:03.414 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:03.059+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:03.414 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:03.062+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:03.415 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:03.062+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:03.415 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:03.062+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:03.415 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:03.063+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:03.670 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:21:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:21:03.611+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9769 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:21:03.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:03.415+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:03.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:03.415+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:03.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:03.415+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:03.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:03.417+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:04.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:04.096+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:04.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:04.096+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:04.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:04.096+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:04.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:04.097+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:04.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:04.093+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:04.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:04.093+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:04.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:04.093+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:04.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:04.094+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:04.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:04.453+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:04.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:04.453+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:04.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:04.453+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:04.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:04.455+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:05.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:05.082+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:05.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:05.082+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:05.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:05.082+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:05.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:05.083+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:05.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:05.120+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:05.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:05.120+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:05.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:05.120+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:05.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:05.120+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:05.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:05.470+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:05.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:05.470+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:05.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:05.470+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:05.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:05.471+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:06.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:06.112+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:06.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:06.112+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:06.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:06.112+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:06.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:06.113+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:06.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:06.109+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:06.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:06.109+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:06.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:06.109+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:06.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:06.110+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:06.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:06.468+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:06.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:06.468+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:06.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:06.468+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:06.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:06.469+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:07.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:07.106+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:07.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:07.106+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:07.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:07.106+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:07.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:07.107+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:07.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:07.151+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:07.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:07.151+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:07.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:07.151+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:07.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:07.152+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:07.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:07.484+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:07.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:07.484+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:07.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:07.484+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:07.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:07.485+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:08.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:08.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:08.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:08.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:08.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:08.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:08.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:08.199+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:08.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:08.083+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:08.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:08.083+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:08.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:08.083+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:08.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:08.084+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:08.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:21:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:21:08.612+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9821 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:21:08.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:08.513+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:08.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:08.513+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:08.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:08.513+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:08.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:08.513+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:09.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:09.216+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:09.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:09.216+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:09.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:09.216+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:09.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:09.217+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:09.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:09.132+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:09.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:09.132+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:09.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:09.132+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:09.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:09.132+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:09.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:09.507+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:09.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:09.507+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:09.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:09.507+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:09.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:09.507+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:10.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:10.134+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:10.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:10.134+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:10.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:10.134+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:10.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:10.135+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:10.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:10.191+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:10.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:10.191+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:10.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:10.191+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:10.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:10.192+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:10.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:10.467+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:10.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:10.467+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:10.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:10.467+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:10.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:10.468+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:11.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:11.200+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:11.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:11.200+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:11.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:11.200+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:11.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:11.201+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:11.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:11.113+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:11.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:11.113+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:11.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:11.113+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:11.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:11.113+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:11.879 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:11.475+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:11.880 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:11.475+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:11.880 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:11.475+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:11.880 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:11.476+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:12.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:12.132+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:12.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:12.132+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:12.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:12.132+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:12.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:12.133+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:12.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:12.170+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:12.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:12.170+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:12.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:12.170+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:12.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:12.171+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:12.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:12.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:12.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:12.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:12.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:12.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:12.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:12.461+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:13.437 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:13.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:13.437 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:13.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:13.437 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:13.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:13.439 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:13.175+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:13.439 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:13.180+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:13.439 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:13.180+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:13.439 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:13.180+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:13.439 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:13.180+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:13.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:21:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:21:13.613+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9872 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:21:13.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:13.439+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:13.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:13.439+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:13.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:13.439+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:13.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:13.439+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:14.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:14.217+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:14.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:14.217+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:14.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:14.217+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:14.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:14.217+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:14.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:14.400+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:14.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:14.400+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:14.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:14.400+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:14.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:14.401+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:14.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:14.155+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:14.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:14.155+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:14.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:14.155+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:14.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:14.156+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:15.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:15.244+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:15.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:15.245+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:15.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:15.245+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:15.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:15.245+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:15.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:15.147+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:15.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:15.147+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:15.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:15.147+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:15.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:15.148+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:15.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:15.439+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:15.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:15.439+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:15.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:15.439+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:15.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:15.440+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:16.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:16.271+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:16.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:16.271+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:16.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:16.271+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:16.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:16.271+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:16.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:16.167+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:16.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:16.167+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:16.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:16.167+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:16.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:16.168+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:16.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:16.457+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:16.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:16.457+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:16.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:16.457+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:16.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:16.458+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:17.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:17.303+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:17.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:17.303+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:17.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:17.303+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:17.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:17.303+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:17.190+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:17.190+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:17.190+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:17.191+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:17.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:17.491+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:17.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:17.491+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:17.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:17.491+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:17.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:17.492+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:18.535 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:18.298+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:18.535 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:18.298+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:18.535 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:18.298+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:18.535 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:18.299+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:18.535 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:18.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:18.535 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:18.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:18.536 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:18.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:18.536 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:18.184+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:18.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:21:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:21:18.614+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9924 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:21:18.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:18.536+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:18.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:18.536+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:18.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:18.536+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:18.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:18.538+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:19.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:19.267+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:19.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:19.267+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:19.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:19.267+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:19.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:19.268+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:19.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:19.140+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:19.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:19.140+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:19.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:19.140+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:19.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:19.141+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:19.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:19.506+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:19.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:19.506+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:19.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:19.506+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:19.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:19.506+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:20.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:20.265+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:20.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:20.265+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:20.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:20.265+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:20.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:20.266+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:20.115+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:20.115+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:20.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:20.115+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:20.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:20.115+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:20.916 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:20.476+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:20.916 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:20.476+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:20.916 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:20.476+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:20.916 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:20.477+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:21.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:21.251+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:21.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:21.251+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:21.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:21.251+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:21.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:21.252+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:21.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:21.103+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:21.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:21.103+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:21.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:21.103+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:21.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:21.104+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:21.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:21.441+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:21.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:21.441+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:21.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:21.441+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:21.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:21.441+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:22.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:22.255+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:22.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:22.255+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:22.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:22.255+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:22.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:22.255+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:22.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:22.104+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:22.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:22.104+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:22.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:22.104+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:22.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:22.105+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:22.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:22.450+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:22.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:22.450+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:22.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:22.450+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:22.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:22.450+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:23.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:23.276+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:23.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:23.276+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:23.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:23.276+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:23.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:23.276+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:23.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:23.116+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:23.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:23.116+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:23.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:23.116+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:23.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:23.117+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:23.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:21:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:21:23.616+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 9974 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:21:23.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:23.459+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:23.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:23.459+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:23.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:23.459+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:23.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:23.460+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:24.240 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:24.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:24.240 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:24.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:24.240 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:24.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:24.240 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:24.089+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:24.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:24.473+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:24.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:24.473+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:24.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:24.473+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:24.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:24.474+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:24.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:24.242+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:24.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:24.242+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:24.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:24.242+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:24.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:24.243+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:25.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:25.100+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:25.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:25.100+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:25.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:25.100+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:25.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:25.101+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:25.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:25.200+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:25.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:25.200+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:25.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:25.200+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:25.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:25.200+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:25.436+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:25.436+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:25.436+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:25.437+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:26.407 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:26.179+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:26.408 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:26.179+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:26.408 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:26.179+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:26.408 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:26.179+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:26.408 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:26.120+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:26.408 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:26.120+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:26.408 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:26.120+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:26.408 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:26.121+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:26.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:26.409+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:26.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:26.409+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:26.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:26.409+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:26.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:26.411+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:27.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:27.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:27.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:27.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:27.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:27.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:27.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:27.183+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:27.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:27.107+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:27.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:27.107+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:27.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:27.107+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:27.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:27.108+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:27.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:27.455+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:27.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:27.455+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:27.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:27.455+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:27.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:27.456+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:28.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:28.116+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:28.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:28.116+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:28.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:28.116+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:28.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:28.117+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:28.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:28.143+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:28.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:28.143+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:28.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:28.143+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:28.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:28.144+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:28.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:21:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:21:28.617+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10024 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:21:28.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:28.434+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:28.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:28.434+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:28.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:28.434+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:28.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:28.435+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:29.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:29.132+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:29.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:29.132+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:29.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:29.132+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:29.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:29.133+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:29.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:29.146+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:29.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:29.146+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:29.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:29.146+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:29.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:29.147+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:29.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:29.477+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:29.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:29.477+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:29.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:29.477+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:29.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:29.478+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:30.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:30.106+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:30.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:30.106+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:30.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:30.106+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:30.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:30.107+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:30.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:30.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:30.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:30.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:30.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:30.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:30.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:30.111+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:30.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:30.453+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:30.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:30.453+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:30.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:30.453+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:30.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:30.454+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:31.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:31.142+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:31.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:31.142+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:31.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:31.142+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:31.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:31.143+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:31.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:31.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:31.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:31.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:31.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:31.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:31.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:31.069+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:31.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:31.450+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:31.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:31.450+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:31.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:31.450+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:31.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:31.451+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:32.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:32.106+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:32.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:32.106+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:32.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:32.106+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:32.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:32.107+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:32.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:32.186+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:32.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:32.186+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:32.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:32.186+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:32.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:32.187+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:32.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:32.457+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:32.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:32.457+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:32.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:32.457+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:32.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:32.458+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:33.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:33.165+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:33.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:33.165+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:33.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:33.165+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:33.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:33.166+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:33.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:33.140+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:33.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:33.140+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:33.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:33.140+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:33.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:33.141+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:33.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:21:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:21:33.618+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10073 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:21:33.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:33.470+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:33.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:33.470+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:33.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:33.470+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:33.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:33.472+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:34.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:34.151+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:34.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:34.151+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:34.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:34.151+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:34.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:34.152+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:34.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:34.189+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:34.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:34.189+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:34.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:34.189+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:34.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:34.191+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:34.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:34.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:34.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:34.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:34.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:34.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:34.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:34.502+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:35.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:35.140+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:35.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:35.140+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:35.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:35.140+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:35.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:35.141+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:35.159+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:35.159+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:35.159+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:35.159+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:35.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:35.456+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:35.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:35.456+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:35.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:35.456+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:35.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:35.457+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:36.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:36.187+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:36.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:36.187+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:36.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:36.187+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:36.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:36.188+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:36.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:36.093+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:36.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:36.093+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:36.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:36.093+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:36.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:36.095+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:36.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:36.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:36.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:36.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:36.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:36.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:36.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:36.490+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:37.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:37.112+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:37.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:37.112+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:37.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:37.112+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:37.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:37.113+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:37.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:37.151+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:37.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:37.151+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:37.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:37.151+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:37.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:37.151+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:37.713 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:37.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:37.714 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:37.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:37.714 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:37.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:37.714 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:37.466+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:38.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:38.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:38.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:38.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:38.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:38.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:38.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:38.111+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:38.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:38.150+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:38.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:38.150+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:38.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:38.150+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:38.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:38.151+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:38.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:21:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:21:38.619+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10123 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:21:38.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:38.458+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:38.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:38.458+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:38.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:38.458+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:38.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:38.459+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:39.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:39.102+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:39.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:39.102+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:39.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:39.102+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:39.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:39.103+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:39.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:39.131+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:39.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:39.131+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:39.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:39.131+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:39.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:39.132+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:39.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:39.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:39.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:39.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:39.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:39.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:39.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:39.490+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:40.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:40.057+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:40.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:40.057+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:40.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:40.057+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:40.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:40.058+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:40.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:40.147+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:40.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:40.147+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:40.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:40.147+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:40.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:40.148+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:40.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:40.481+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:40.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:40.481+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:40.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:40.481+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:40.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:40.482+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:41.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:41.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:41.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:41.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:41.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:41.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:41.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:41.030+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:41.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:41.104+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:41.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:41.104+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:41.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:41.104+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:41.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:41.105+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:41.908 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:41.454+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:41.908 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:41.454+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:41.908 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:41.454+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:41.908 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:41.455+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:42.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:42.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:42.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:42.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:42.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:42.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:42.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:42.079+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:42.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:42.154+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:42.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:42.154+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:42.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:42.154+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:42.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:42.155+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:42.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:42.483+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:42.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:42.483+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:42.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:42.483+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:42.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:42.484+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:43.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:43.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:43.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:43.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:43.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:43.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:43.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:43.033+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:43.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:43.132+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:43.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:43.132+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:43.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:43.132+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:43.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:43.133+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:43.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:21:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:21:43.620+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10173 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:21:43.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:43.519+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:43.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:43.519+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:43.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:43.519+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:43.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:43.520+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:44.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:44.153+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:44.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:44.153+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:44.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:44.153+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:44.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:44.153+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:44.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:44.043+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:44.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:44.043+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:44.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:44.043+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:44.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:44.043+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:44.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:44.535+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:44.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:44.535+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:44.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:44.535+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:44.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:44.535+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:45.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:45.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:45.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:45.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:45.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:45.070+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:45.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:45.167+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:45.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:45.167+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:45.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:45.167+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:45.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:45.168+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:45.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:45.576+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:45.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:45.576+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:45.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:45.576+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:45.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:45.576+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:46.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:46.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:46.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:46.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:46.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:46.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:46.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:46.082+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:46.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:46.178+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:46.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:46.178+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:46.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:46.178+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:46.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:46.179+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:46.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:46.533+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:46.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:46.533+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:46.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:46.533+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:46.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:46.534+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:47.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:47.161+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:47.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:47.161+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:47.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:47.161+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:47.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:47.162+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:47.093+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:47.093+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:47.093+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:47.093+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:47.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:47.528+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:47.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:47.528+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:47.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:47.528+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:47.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:47.529+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:48.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:48.164+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:48.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:48.164+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:48.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:48.164+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:48.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:48.164+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:48.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:48.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:48.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:48.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:48.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:48.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:48.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:48.064+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:48.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:21:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:21:48.621+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10224 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:21:48.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:48.566+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:48.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:48.566+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:48.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:48.566+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:48.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:48.567+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:49.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:49.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:49.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:49.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:49.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:49.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:49.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:49.070+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:49.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:49.185+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:49.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:49.185+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:49.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:49.185+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:49.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:49.186+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:49.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:49.586+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:49.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:49.586+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:49.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:49.586+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:49.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:49.588+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:50.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:50.085+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:50.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:50.085+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:50.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:50.085+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:50.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:50.085+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:50.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:50.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:50.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:50.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:50.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:50.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:50.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:50.182+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:50.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:50.622+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:50.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:50.622+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:50.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:50.622+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:50.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:50.622+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:51.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:51.096+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:51.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:51.096+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:51.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:51.096+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:51.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:51.097+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:51.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:51.229+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:51.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:51.229+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:51.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:51.229+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:51.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:51.230+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:51.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:51.591+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:51.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:51.591+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:51.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:51.591+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:51.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:51.592+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:52.371 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:52.244+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:52.371 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:52.244+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:52.371 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:52.244+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:52.371 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:52.245+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:52.371 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:52.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:52.372 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:52.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:52.372 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:52.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:52.372 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:52.090+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:52.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:52.561+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:52.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:52.561+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:52.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:52.561+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:52.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:52.562+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:53.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:53.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:53.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:53.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:53.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:53.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:53.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:53.062+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:53.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:53.293+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:53.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:53.293+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:53.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:53.293+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:53.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:53.293+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:53.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:21:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:21:53.622+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10274 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:21:53.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:53.536+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:53.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:53.536+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:53.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:53.536+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:53.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:53.537+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:54.337 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:54.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:54.337 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:54.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:54.337 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:54.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:54.337 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:54.065+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:54.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:54.573+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:54.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:54.573+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:54.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:54.573+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:54.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:54.574+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:54.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:54.339+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:54.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:54.339+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:54.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:54.339+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:54.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:54.340+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:55.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:55.108+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:55.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:55.108+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:55.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:55.108+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:55.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:55.109+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:55.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:55.336+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:55.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:55.336+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:55.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:55.336+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:55.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:55.337+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:55.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:55.584+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:55.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:55.584+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:55.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:55.584+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:55.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:55.585+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:56.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:56.127+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:56.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:56.127+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:56.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:56.127+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:56.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:56.127+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:56.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:56.295+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:56.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:56.295+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:56.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:56.295+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:56.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:56.296+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:56.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:56.622+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:56.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:56.622+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:56.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:56.622+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:56.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:56.622+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:57.593 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:57.257+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:57.593 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:57.257+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:57.593 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:57.257+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:57.593 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:57.258+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:57.593 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:57.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:57.593 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:57.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:57.593 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:57.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:57.593 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:57.176+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:57.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:57.594+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:57.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:57.594+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:57.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:57.594+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:57.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:57.596+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:58.248 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:58.138+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:58.248 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:58.138+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:58.248 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:58.138+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:58.248 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:58.139+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:58.622 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:58.250+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:58.622 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:58.250+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:58.622 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:58.250+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:58.622 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:58.251+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:58.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:21:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:21:58.624+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10325 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:21:58.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:58.640+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:58.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:58.641+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:58.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:58.641+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:58.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:58.641+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:59.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:59.112+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:59.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:59.112+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:59.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:59.112+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:21:59.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:21:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:21:59.113+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:21:59.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:59.262+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:59.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:59.262+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:59.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:59.262+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:21:59.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:21:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:21:59.263+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:21:59.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:59.641+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:59.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:59.641+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:59.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:59.641+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:21:59.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:21:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:21:59.641+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:00.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:00.286+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:00.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:00.286+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:00.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:00.286+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:00.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:00.288+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:00.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:00.158+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:00.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:00.158+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:00.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:00.158+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:00.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:00.160+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:00.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:00.621+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:00.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:00.621+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:00.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:00.621+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:00.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:00.622+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:01.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:01.191+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:01.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:01.191+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:01.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:01.191+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:01.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:01.191+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:01.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:01.304+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:01.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:01.304+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:01.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:01.304+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:01.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:01.305+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:01.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:01.663+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:01.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:01.663+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:01.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:01.663+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:01.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:01.664+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:02.667 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:02.286+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:02.667 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:02.286+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:02.667 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:02.286+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:02.667 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:02.287+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:02.667 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:02.220+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:02.668 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:02.220+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:02.668 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:02.220+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:02.668 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:02.221+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:02.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:02.669+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:02.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:02.669+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:02.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:02.669+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:02.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:02.670+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:03.578 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:03.287+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:03.578 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:03.287+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:03.578 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:03.287+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:03.579 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:03.287+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:03.579 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:03.192+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:03.579 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:03.192+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:03.579 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:03.192+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:03.579 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:03.193+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:03.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:22:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:22:03.625+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10375 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:22:03.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:03.718+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:03.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:03.718+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:03.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:03.718+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:03.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:03.720+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:04.282 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:04.173+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:04.283 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:04.173+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:04.283 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:04.173+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:04.283 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:04.174+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:04.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:04.284+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:04.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:04.284+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:04.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:04.284+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:04.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:04.285+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:05.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:04.729+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:05.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:04.729+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:05.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:04.729+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:05.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:04.729+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:05.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:05.193+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:05.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:05.193+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:05.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:05.193+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:05.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:05.194+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:05.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:05.260+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:05.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:05.260+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:05.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:05.260+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:05.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:05.261+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:06.143 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:05.715+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:06.143 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:05.715+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:06.143 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:05.715+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:06.143 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:05.715+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:06.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:06.212+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:06.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:06.212+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:06.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:06.212+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:06.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:06.213+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:06.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:06.145+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:06.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:06.145+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:06.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:06.145+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:06.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:06.146+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:07.109 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:06.754+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:07.109 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:06.754+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:07.109 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:06.754+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:07.109 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:06.755+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:07.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:07.263+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:07.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:07.263+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:07.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:07.263+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:07.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:07.263+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:07.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:07.111+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:07.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:07.111+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:07.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:07.111+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:07.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:07.112+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:08.067 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:07.786+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:08.067 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:07.786+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:08.067 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:07.786+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:08.067 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:07.787+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:08.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:08.311+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:08.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:08.311+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:08.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:08.311+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:08.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:08.312+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:08.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:08.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:08.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:08.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:08.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:08.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:08.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:08.069+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:08.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:22:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:22:08.626+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10426 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:22:08.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:08.789+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:08.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:08.789+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:08.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:08.789+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:08.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:08.790+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:09.348 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:09.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:09.348 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:09.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:09.348 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:09.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:09.348 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:09.051+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:09.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:09.349+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:09.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:09.349+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:09.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:09.349+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:09.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:09.351+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:10.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:10.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:10.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:10.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:10.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:10.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:10.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:10.003+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:10.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:09.822+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:10.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:09.822+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:10.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:09.822+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:10.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:09.823+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:10.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:10.371+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:10.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:10.371+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:10.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:10.371+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:10.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:10.372+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:11.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:10.966+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:11.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:10.966+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:11.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:10.966+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:11.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:10.966+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:11.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:10.781+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:11.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:10.781+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:11.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:10.781+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:11.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:10.782+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:11.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:11.341+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:11.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:11.341+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:11.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:11.341+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:11.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:11.342+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:12.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:11.998+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:12.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:11.998+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:12.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:11.998+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:12.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:11.999+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:12.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:11.818+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:12.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:11.818+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:12.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:11.818+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:12.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:11.818+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:12.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:12.342+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:12.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:12.342+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:12.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:12.342+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:12.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:12.343+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:13.140 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:13.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:13.140 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:13.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:13.140 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:13.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:13.140 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:13.036+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:13.140 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:12.803+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:13.140 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:12.803+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:13.140 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:12.803+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:13.140 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:12.803+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:13.579 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:13.301+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:13.579 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:13.301+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:13.580 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:13.301+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:13.580 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:13.301+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:13.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:22:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:22:13.626+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10475 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:22:13.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:13.806+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:13.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:13.806+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:13.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:13.806+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:13.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:13.806+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:14.319 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:14.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:14.320 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:14.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:14.320 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:14.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:14.320 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:14.016+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:14.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:14.320+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:14.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:14.320+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:14.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:14.320+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:14.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:14.321+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:15.064 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:14.804+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:15.064 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:14.804+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:15.064 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:14.804+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:15.064 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:14.804+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:15.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:15.065+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:15.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:15.065+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:15.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:15.065+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:15.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:15.066+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:15.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:15.292+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:15.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:15.292+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:15.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:15.292+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:15.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:15.293+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:16.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:16.046+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:16.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:16.046+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:16.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:16.046+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:16.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:16.047+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:16.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:15.825+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:16.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:15.825+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:16.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:15.825+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:16.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:15.826+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:16.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:16.287+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:16.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:16.287+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:16.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:16.287+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:16.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:16.288+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:17.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:16.794+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:17.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:16.794+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:17.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:16.794+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:17.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:16.794+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:17.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:16.999+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:17.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:16.999+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:17.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:16.999+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:17.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:17.000+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:17.560 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:17.257+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:17.560 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:17.257+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:17.560 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:17.257+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:17.560 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:17.258+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:17.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:17.750+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:17.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:17.750+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:17.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:17.750+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:17.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:17.750+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:18.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:18.220+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:18.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:18.220+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:18.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:18.220+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:18.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:18.221+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:17.984+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:17.984+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:17.984+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:17.985+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:18.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:22:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:22:18.627+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10525 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:22:18.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:18.706+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:18.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:18.706+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:18.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:18.706+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:18.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:18.707+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:19.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:19.003+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:19.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:19.003+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:19.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:19.003+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:19.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:19.004+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:19.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:19.248+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:19.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:19.248+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:19.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:19.248+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:19.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:19.249+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:19.996 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:19.691+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:19.996 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:19.691+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:19.996 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:19.691+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:19.996 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:19.692+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:20.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:20.203+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:20.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:20.203+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:20.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:20.203+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:20.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:20.205+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:19.996+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:19.996+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:19.996+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:19.997+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:20.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:20.671+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:20.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:20.671+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:20.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:20.671+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:20.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:20.672+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:21.203 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:20.952+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:21.203 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:20.952+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:21.203 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:20.952+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:21.204 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:20.953+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:21.568 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:21.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:21.568 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:21.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:21.568 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:21.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:21.568 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:21.205+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:21.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:21.685+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:21.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:21.685+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:21.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:21.685+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:21.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:21.685+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:22.226 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:21.940+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:22.226 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:21.940+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:22.226 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:21.940+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:22.226 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:21.942+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:22.597 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:22.228+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:22.597 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:22.228+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:22.597 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:22.228+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:22.597 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:22.228+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:22.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:22.722+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:22.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:22.722+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:22.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:22.722+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:22.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:22.723+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:23.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:22.965+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:23.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:22.965+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:23.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:22.965+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:23.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:22.966+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:23.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:23.206+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:23.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:23.206+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:23.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:23.206+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:23.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:23.206+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:23.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:22:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:22:23.628+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10574 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:22:23.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:23.746+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:23.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:23.746+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:23.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:23.746+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:23.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:23.746+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:24.238 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:23.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:24.238 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:23.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:24.238 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:23.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:24.238 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:23.987+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:24.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:24.240+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:24.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:24.240+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:24.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:24.240+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:24.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:24.241+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:25.014 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:24.720+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:25.014 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:24.720+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:25.015 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:24.720+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:25.015 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:24.721+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:25.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:25.193+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:25.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:25.193+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:25.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:25.193+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:25.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:25.194+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:25.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:25.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:25.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:25.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:25.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:25.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:25.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:25.017+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:26.040 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:25.755+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:26.040 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:25.756+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:26.040 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:25.756+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:26.040 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:25.756+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:26.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:26.216+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:26.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:26.216+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:26.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:26.216+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:26.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:26.217+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:26.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:26.042+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:26.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:26.042+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:26.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:26.042+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:26.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:26.043+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:27.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:26.798+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:27.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:26.798+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:27.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:26.798+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:27.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:26.800+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:27.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:27.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:27.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:27.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:27.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:27.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:27.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:27.026+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:27.641 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:27.198+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:27.641 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:27.198+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:27.641 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:27.198+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:27.641 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:27.198+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:27.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:27.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:27.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:27.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:27.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:27.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:27.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:27.846+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:28.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:28.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:28.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:28.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:28.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:28.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:28.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:28.204+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:28.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:28.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:28.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:28.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:28.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:28.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:28.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:28.071+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:28.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:22:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:22:28.630+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10624 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:22:28.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:28.802+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:28.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:28.802+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:28.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:28.802+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:28.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:28.803+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:29.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:29.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:29.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:29.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:29.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:29.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:29.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:29.075+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:29.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:29.202+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:29.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:29.202+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:29.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:29.202+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:29.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:29.203+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:30.106 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:29.798+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:30.106 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:29.798+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:30.106 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:29.798+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:30.106 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:29.800+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:30.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:30.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:30.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:30.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:30.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:30.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:30.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:30.106+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:30.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:30.245+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:30.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:30.245+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:30.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:30.245+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:30.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:30.246+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:31.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:30.830+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:31.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:30.830+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:31.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:30.830+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:31.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:30.831+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:31.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:31.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:31.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:31.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:31.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:31.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:31.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:31.064+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:31.567 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:31.206+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:31.567 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:31.206+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:31.568 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:31.206+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:31.568 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:31.207+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:31.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:31.804+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:31.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:31.804+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:31.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:31.804+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:31.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:31.805+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:32.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:32.185+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:32.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:32.185+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:32.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:32.185+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:32.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:32.186+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:32.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:32.021+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:32.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:32.021+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:32.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:32.021+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:32.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:32.022+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:33.064 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:32.787+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:33.065 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:32.787+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:33.065 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:32.787+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:33.065 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:32.788+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:33.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:33.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:33.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:33.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:33.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:33.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:33.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:33.067+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:33.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:33.172+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:33.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:33.172+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:33.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:33.172+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:33.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:33.173+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:33.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:22:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:22:33.631+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10673 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:22:33.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:33.751+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:33.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:33.751+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:33.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:33.751+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:33.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:33.752+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:34.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:34.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:34.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:34.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:34.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:34.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:34.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:34.073+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:34.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:34.150+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:34.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:34.150+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:34.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:34.150+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:34.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:34.151+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:34.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:34.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:34.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:34.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:34.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:34.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:34.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:34.775+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:35.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:35.197+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:35.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:35.197+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:35.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:35.197+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:35.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:35.198+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:35.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:35.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:35.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:35.065+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:36.076 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:35.806+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:36.076 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:35.806+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:36.076 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:35.806+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:36.077 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:35.807+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:36.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:36.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:36.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:36.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:36.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:36.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:36.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:36.079+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:36.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:36.220+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:36.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:36.220+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:36.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:36.220+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:36.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:36.221+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:37.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:36.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:37.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:36.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:37.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:36.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:37.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:36.846+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:37.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:37.084+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:37.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:37.084+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:37.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:37.084+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:37.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:37.085+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:37.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:37.172+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:37.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:37.172+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:37.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:37.172+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:37.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:37.173+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:38.073 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:37.830+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:38.074 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:37.830+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:38.074 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:37.830+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:38.074 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:37.832+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:38.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:38.156+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:38.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:38.156+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:38.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:38.156+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:38.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:38.157+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:38.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:38.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:38.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:38.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:38.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:38.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:38.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:38.077+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:38.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:22:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:22:38.632+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10723 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:22:38.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:38.785+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:38.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:38.785+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:38.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:38.785+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:38.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:38.787+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:39.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:39.135+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:39.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:39.135+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:39.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:39.135+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:39.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:39.136+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:39.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:39.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:39.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:39.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:39.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:39.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:39.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:39.037+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:40.126 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:39.752+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:40.126 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:39.752+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:40.126 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:39.752+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:40.126 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:39.753+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:40.127 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:40.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:40.127 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:40.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:40.127 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:40.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:40.127 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:40.001+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:40.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:40.127+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:40.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:40.127+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:40.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:40.127+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:40.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:40.128+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:41.104 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:40.759+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:41.104 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:40.759+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:41.104 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:40.759+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:41.104 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:40.760+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:41.105 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:40.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:41.105 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:40.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:41.105 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:40.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:41.105 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:40.993+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:41.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:41.106+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:41.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:41.106+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:41.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:41.106+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:41.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:41.107+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:41.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:41.759+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:41.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:41.759+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:41.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:41.759+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:41.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:41.760+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:42.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:42.071+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:42.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:42.071+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:42.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:42.071+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:42.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:42.072+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:42.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:41.995+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:42.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:41.995+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:42.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:41.995+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:42.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:41.996+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:43.037 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:42.987+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:43.037 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:42.987+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:43.038 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:42.987+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:43.038 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:42.988+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:43.038 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:42.787+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:43.038 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:42.787+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:43.038 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:42.787+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:43.038 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:42.788+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:43.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:43.039+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:43.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:43.039+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:43.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:43.039+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:43.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:43.040+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:43.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:22:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:22:43.633+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10772 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:22:43.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:43.745+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:43.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:43.745+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:43.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:43.745+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:43.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:43.746+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:44.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:44.009+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:44.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:44.009+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:44.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:44.009+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:44.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:44.010+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:44.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:44.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:44.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:44.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:44.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:44.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:44.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:44.072+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:45.021 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:44.747+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:45.021 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:44.747+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:45.022 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:44.747+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:45.022 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:44.749+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:45.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:45.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:45.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:45.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:45.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:45.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:45.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:45.024+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:45.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:45.088+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:45.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:45.088+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:45.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:45.088+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:45.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:45.089+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:46.069 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:45.781+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:46.069 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:45.781+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:46.069 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:45.781+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:46.069 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:45.783+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:46.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:46.088+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:46.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:46.088+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:46.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:46.088+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:46.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:46.088+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:46.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:46.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:46.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:46.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:46.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:46.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:46.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:46.071+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:47.079 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:46.794+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:47.079 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:46.794+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:47.079 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:46.794+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:47.079 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:46.795+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:47.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:47.115+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:47.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:47.115+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:47.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:47.115+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:47.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:47.116+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:47.080+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:47.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:47.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:47.081+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:48.099 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:47.807+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:48.099 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:47.807+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:48.099 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:47.807+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:48.099 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:47.808+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:48.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:48.101+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:48.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:48.101+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:48.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:48.101+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:48.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:48.101+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:48.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:48.112+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:48.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:48.112+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:48.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:48.112+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:48.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:48.113+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:48.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:22:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:22:48.634+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10822 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:22:48.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:48.835+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:48.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:48.835+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:48.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:48.835+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:48.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:48.836+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:49.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:49.146+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:49.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:49.146+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:49.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:49.146+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:49.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:49.146+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:49.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:49.131+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:49.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:49.131+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:49.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:49.131+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:49.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:49.132+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:50.110 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:49.848+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:50.111 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:49.848+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:50.111 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:49.848+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:50.111 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:49.848+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:50.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:50.187+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:50.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:50.187+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:50.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:50.187+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:50.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:50.188+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:50.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:50.112+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:50.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:50.112+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:50.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:50.112+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:50.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:50.112+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:51.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:51.094+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:51.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:51.094+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:51.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:51.094+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:51.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:51.095+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:51.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:50.896+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:51.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:50.896+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:51.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:50.896+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:51.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:50.896+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:51.568 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:51.199+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:51.568 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:51.199+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:51.568 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:51.199+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:51.568 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:51.200+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:52.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:52.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:52.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:52.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:52.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:52.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:52.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:52.070+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:52.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:51.850+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:52.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:51.850+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:52.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:51.850+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:52.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:51.850+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:52.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:52.174+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:52.670 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:52.174+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:52.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:52.174+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:52.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:52.174+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:53.131 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:53.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:53.131 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:53.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:53.131 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:53.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:53.131 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:53.087+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:53.131 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:52.856+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:53.131 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:52.856+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:53.131 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:52.856+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:53.131 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:52.857+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:53.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:53.132+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:53.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:53.133+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:53.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:53.133+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:53.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:53.133+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:53.883 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:22:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:22:53.636+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10871 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:22:54.142 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:53.885+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:54.142 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:53.885+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:54.142 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:53.885+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:54.142 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:53.885+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:54.142 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:54.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:54.143 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:54.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:54.143 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:54.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:54.143 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:54.133+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:54.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:54.144+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:54.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:54.144+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:54.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:54.144+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:54.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:54.145+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:55.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:54.905+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:55.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:54.905+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:55.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:54.905+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:55.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:54.907+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:55.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:55.118+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:55.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:55.118+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:55.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:55.118+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:55.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:55.119+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:55.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:55.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:55.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:55.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:55.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:55.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:55.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:55.184+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:56.147 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:55.873+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:56.147 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:55.873+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:56.147 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:55.873+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:56.147 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:55.874+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:56.148 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:56.107+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:56.148 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:56.107+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:56.148 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:56.107+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:56.148 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:56.108+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:56.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:56.157+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:56.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:56.157+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:56.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:56.157+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:56.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:56.158+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:57.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:56.868+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:57.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:56.868+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:57.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:56.868+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:57.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:56.868+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:57.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:57.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:57.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:57.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:57.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:57.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:57.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:57.073+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:57.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:57.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:57.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:57.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:57.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:57.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:57.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:57.172+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:58.160 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:57.843+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:58.160 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:57.843+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:58.161 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:57.843+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:58.161 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:57.844+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:58.161 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:58.050+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:58.161 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:58.050+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:58.161 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:58.050+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:58.161 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:58.051+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:58.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:58.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:58.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:58.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:58.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:58.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:58.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:58.163+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:58.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:22:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:22:58.637+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10921 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:22:58.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:58.844+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:58.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:58.844+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:58.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:58.844+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:22:58.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:58.845+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:22:59.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:59.053+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:59.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:59.053+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:59.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:59.053+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:22:59.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:22:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:22:59.054+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:22:59.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:59.121+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:59.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:59.121+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:59.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:59.121+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:22:59.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:22:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:22:59.122+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:00.039 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:59.887+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:00.039 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:59.887+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:00.039 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:59.887+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:00.039 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:22:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:22:59.888+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:00.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:00.041+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:00.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:00.041+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:00.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:00.041+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:00.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:00.042+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:00.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:00.087+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:00.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:00.087+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:00.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:00.087+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:00.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:00.088+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:01.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:01.010+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:01.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:01.010+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:01.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:01.010+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:01.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:01.010+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:01.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:01.075+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:01.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:01.075+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:01.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:01.075+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:01.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:01.076+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:01.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:00.891+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:01.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:00.891+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:01.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:00.891+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:01.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:00.892+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:02.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:01.856+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:02.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:01.856+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:02.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:01.856+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:02.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:01.857+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:02.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:02.016+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:02.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:02.016+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:02.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:02.016+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:02.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:02.018+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:02.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:02.037+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:02.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:02.037+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:02.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:02.037+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:02.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:02.038+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:03.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:02.897+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:03.170 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:02.897+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:03.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:02.897+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:03.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:02.898+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:03.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:03.073+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:03.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:03.073+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:03.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:03.073+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:03.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:03.073+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:03.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:03.023+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:03.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:03.023+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:03.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:03.023+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:03.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:03.024+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:03.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:23:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:23:03.638+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 10970 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:23:04.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:04.010+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:04.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:04.010+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:04.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:04.010+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:04.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:04.011+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:04.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:03.947+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:04.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:03.947+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:04.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:03.947+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:04.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:03.948+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:04.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:04.041+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:04.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:04.041+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:04.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:04.041+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:04.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:04.041+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:05.363 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:04.998+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:05.363 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:04.998+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:05.363 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:04.998+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:05.363 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:04.998+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:05.364 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:04.934+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:05.364 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:04.934+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:05.364 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:04.934+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:05.364 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:04.936+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:05.364 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:05.059+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:05.364 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:05.059+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:05.364 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:05.059+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:05.364 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:05.059+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:06.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:06.054+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:06.054+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:06.054+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:06.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:06.054+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:06.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:05.918+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:06.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:05.918+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:06.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:05.918+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:06.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:05.918+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:06.032+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:06.032+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:06.032+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:06.032+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:07.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:07.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:07.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:07.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:07.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:07.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:07.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:07.064+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:07.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:06.950+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:07.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:06.950+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:07.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:06.950+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:07.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:06.951+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:07.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:07.056+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:07.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:07.056+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:07.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:07.056+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:07.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:07.056+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:08.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:08.109+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:08.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:08.109+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:08.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:08.109+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:08.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:08.109+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:08.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:07.988+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:08.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:07.988+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:08.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:07.988+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:08.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:07.990+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:08.020+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:08.020+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:08.020+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:08.021+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:08.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:23:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:23:08.639+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11020 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:23:09.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:09.082+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:09.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:09.082+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:09.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:09.082+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:09.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:09.082+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:09.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:09.004+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:09.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:09.004+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:09.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:09.004+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:09.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:09.005+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:09.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:08.975+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:09.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:08.975+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:09.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:08.975+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:09.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:08.976+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:10.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:10.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:10.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:10.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:10.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:10.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:10.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:10.080+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:10.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:10.050+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:10.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:10.050+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:10.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:10.050+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:10.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:10.051+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:10.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:09.941+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:10.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:09.941+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:10.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:09.941+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:10.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:09.942+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:11.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:11.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:11.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:11.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:11.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:11.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:11.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:11.052+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:11.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:11.056+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:11.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:11.056+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:11.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:11.056+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:11.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:11.057+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:11.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:10.943+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:11.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:10.943+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:11.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:10.943+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:11.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:10.944+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:12.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:12.073+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:12.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:12.073+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:12.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:12.073+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:12.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:12.074+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:12.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:12.075+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:12.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:12.075+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:12.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:12.075+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:12.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:12.076+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:12.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:11.982+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:12.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:11.982+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:12.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:11.982+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:12.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:11.984+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:13.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:13.053+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:13.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:13.053+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:13.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:13.053+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:13.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:13.054+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:13.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:13.118+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:13.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:13.118+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:13.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:13.118+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:13.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:13.119+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:13.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:13.029+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:13.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:13.029+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:13.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:13.029+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:13.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:13.030+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:13.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:23:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:23:13.640+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11069 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:23:14.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:14.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:14.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:14.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:14.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:14.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:14.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:14.034+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:14.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:14.128+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:14.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:14.128+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:14.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:14.128+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:14.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:14.128+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:14.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:14.038+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:14.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:14.038+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:14.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:14.038+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:14.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:14.039+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:15.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:15.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:15.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:15.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:15.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:15.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:15.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:15.072+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:15.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:15.112+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:15.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:15.112+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:15.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:15.112+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:15.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:15.113+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:15.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:15.063+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:15.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:15.063+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:15.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:15.063+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:15.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:15.063+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:16.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:16.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:16.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:16.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:16.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:16.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:16.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:16.082+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:16.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:16.154+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:16.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:16.154+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:16.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:16.154+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:16.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:16.154+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:16.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:16.066+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:16.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:16.066+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:16.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:16.066+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:16.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:16.067+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:17.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:17.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:17.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:17.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:17.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:17.035+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:17.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:17.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:17.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:17.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:17.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:17.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:17.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:17.189+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:17.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:17.082+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:17.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:17.082+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:17.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:17.082+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:17.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:17.083+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:18.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:18.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:18.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:18.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:18.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:18.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:18.071+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:18.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:18.201+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:18.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:18.201+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:18.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:18.201+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:18.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:18.202+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:18.097+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:18.097+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:18.097+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:18.098+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:18.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:23:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:23:18.641+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11119 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:23:19.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:19.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:19.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:19.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:19.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:19.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:19.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:19.070+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:19.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:19.154+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:19.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:19.154+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:19.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:19.154+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:19.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:19.155+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:19.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:19.066+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:19.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:19.066+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:19.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:19.066+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:19.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:19.068+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:20.056+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:20.056+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:20.056+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:20.057+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:20.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:20.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:20.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:20.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:20.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:20.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:20.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:20.142+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:20.047+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:20.047+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:20.047+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:20.048+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:21.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:21.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:21.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:21.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:21.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:21.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:21.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:21.072+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:21.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:21.127+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:21.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:21.127+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:21.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:21.127+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:21.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:21.128+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:21.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:21.052+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:21.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:21.052+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:21.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:21.052+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:21.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:21.053+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:22.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:22.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:22.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:22.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:22.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:22.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:22.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:22.026+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:22.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:22.107+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:22.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:22.107+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:22.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:22.107+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:22.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:22.108+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:22.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:22.037+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:22.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:22.037+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:22.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:22.037+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:22.038+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:23.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:23.006+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:23.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:23.006+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:23.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:23.006+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:23.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:23.007+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:23.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:23.133+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:23.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:23.133+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:23.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:23.133+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:23.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:23.133+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:23.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:22.991+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:23.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:22.991+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:23.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:22.991+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:23.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:22.992+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:23.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:23:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:23:23.642+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11168 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:23:24.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:24.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:24.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:24.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:24.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:24.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:24.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:24.034+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:24.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:23.994+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:24.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:23.994+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:24.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:23.994+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:24.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:23.995+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:24.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:24.085+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:24.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:24.085+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:24.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:24.085+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:24.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:24.086+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:25.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:25.074+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:25.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:25.074+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:25.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:25.074+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:25.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:25.075+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:25.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:25.066+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:25.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:25.066+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:25.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:25.066+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:25.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:25.067+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:25.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:25.021+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:25.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:25.021+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:25.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:25.021+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:25.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:25.021+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:26.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:26.027+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:26.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:26.027+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:26.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:26.027+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:26.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:26.028+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:26.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:26.070+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:26.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:26.070+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:26.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:26.070+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:26.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:26.072+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:26.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:25.981+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:26.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:25.981+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:26.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:25.981+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:26.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:25.982+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:27.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:27.018+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:27.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:27.018+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:27.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:27.018+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:27.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:27.018+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:27.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:27.033+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:27.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:27.033+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:27.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:27.033+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:27.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:27.034+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:27.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:27.024+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:27.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:27.024+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:27.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:27.024+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:27.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:27.024+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:28.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:28.044+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:28.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:28.044+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:28.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:28.044+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:28.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:28.045+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:28.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:28.026+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:28.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:28.026+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:28.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:28.026+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:28.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:28.028+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:28.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:28.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:28.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:28.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:28.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:28.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:28.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:28.007+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:28.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:23:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:23:28.643+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11218 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:23:29.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:29.038+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:29.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:29.038+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:29.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:29.038+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:29.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:29.039+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:29.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:28.998+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:29.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:28.998+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:29.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:28.998+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:29.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:28.999+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:29.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:29.008+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:29.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:29.008+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:29.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:29.008+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:29.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:29.008+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:30.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:30.070+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:30.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:30.070+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:30.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:30.070+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:30.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:30.071+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:30.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:29.989+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:30.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:29.989+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:30.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:29.989+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:30.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:29.990+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:30.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:30.039+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:30.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:30.039+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:30.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:30.039+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:30.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:30.040+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:31.321 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:31.014+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:31.321 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:31.014+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:31.321 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:31.014+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:31.321 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:31.014+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:31.322 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:31.086+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:31.322 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:31.086+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:31.322 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:31.086+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:31.322 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:31.087+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:31.322 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:30.996+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:31.322 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:30.996+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:31.322 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:30.996+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:31.322 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:30.996+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:32.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:31.989+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:32.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:31.989+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:32.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:31.989+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:32.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:31.990+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:32.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:32.041+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:32.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:32.041+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:32.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:32.041+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:32.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:32.042+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:32.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:32.128+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:32.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:32.128+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:32.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:32.128+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:32.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:32.129+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:33.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:32.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:33.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:32.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:33.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:32.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:33.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:32.995+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:33.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:33.147+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:33.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:33.147+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:33.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:33.147+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:33.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:33.148+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:33.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:33.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:33.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:33.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:33.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:33.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:33.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:33.079+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:33.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:23:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:23:33.645+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11267 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:23:34.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:34.065+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:34.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:34.065+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:34.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:34.065+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:34.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:34.066+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:34.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:34.107+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:34.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:34.107+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:34.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:34.107+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:34.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:34.108+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:34.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:33.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:34.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:33.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:34.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:33.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:34.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:33.974+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:35.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:35.006+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:35.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:35.006+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:35.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:35.006+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:35.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:35.007+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:35.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:35.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:35.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:35.025+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:35.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:35.081+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:35.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:35.081+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:35.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:35.081+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:35.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:35.082+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:36.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:36.021+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:36.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:36.021+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:36.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:36.021+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:36.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:36.022+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:36.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:36.026+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:36.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:36.026+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:36.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:36.026+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:36.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:36.026+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:36.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:36.039+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:36.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:36.039+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:36.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:36.039+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:36.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:36.040+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:37.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:36.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:37.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:36.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:37.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:36.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:37.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:36.979+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:37.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:36.997+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:37.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:36.997+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:37.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:36.997+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:37.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:36.998+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:37.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:37.015+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:37.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:37.015+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:37.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:37.015+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:37.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:37.016+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:38.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:37.993+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:38.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:37.993+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:38.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:37.993+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:38.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:37.994+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:38.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:37.970+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:38.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:37.970+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:38.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:37.970+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:38.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:37.970+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:38.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:37.982+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:38.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:37.982+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:38.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:37.982+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:38.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:37.983+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:38.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:23:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:23:38.646+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11317 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:23:39.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:39.029+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:39.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:39.029+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:39.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:39.029+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:39.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:39.030+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:39.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:38.965+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:39.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:38.965+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:39.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:38.965+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:39.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:38.967+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:39.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:38.940+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:39.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:38.940+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:39.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:38.940+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:39.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:38.941+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:40.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:39.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:40.170 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:39.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:40.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:39.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:40.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:39.963+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:40.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:40.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:40.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:40.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:40.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:40.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:40.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:40.048+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:40.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:39.922+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:40.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:39.922+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:40.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:39.922+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:40.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:39.922+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:41.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:40.970+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:41.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:40.970+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:41.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:40.970+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:41.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:40.971+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:41.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:40.914+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:41.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:40.914+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:41.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:40.914+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:41.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:40.915+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:41.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:41.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:41.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:41.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:41.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:41.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:41.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:41.041+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:42.228 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:42.068+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:42.228 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:42.068+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:42.228 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:42.068+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:42.228 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:42.069+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:42.229 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:41.949+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:42.229 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:41.949+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:42.229 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:41.949+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:42.229 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:41.950+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:42.229 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:41.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:42.229 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:41.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:42.229 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:41.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:42.229 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:41.962+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:43.241 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:43.034+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:43.241 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:43.034+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:43.241 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:43.034+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:43.241 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:43.035+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:43.241 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:42.969+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:43.242 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:42.969+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:43.242 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:42.969+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:43.242 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:42.970+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:43.242 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:42.947+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:43.242 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:42.947+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:43.242 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:42.947+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:43.242 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:42.948+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:43.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:23:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:23:43.646+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11366 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:23:44.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:43.995+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:44.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:43.995+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:44.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:43.995+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:44.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:43.997+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:44.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:43.970+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:44.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:43.970+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:44.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:43.970+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:44.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:43.971+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:44.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:44.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:44.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:44.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:44.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:44.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:44.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:44.023+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:45.299 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:44.998+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:45.299 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:44.998+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:45.299 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:44.998+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:45.300 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:44.999+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:45.300 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:45.068+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:45.300 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:45.068+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:45.300 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:45.068+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:45.300 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:45.069+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:45.300 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:44.998+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:45.300 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:44.998+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:45.301 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:44.998+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:45.301 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:44.999+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:46.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:46.014+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:46.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:46.014+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:46.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:46.014+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:46.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:46.015+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:46.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:46.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:46.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:46.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:46.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:46.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:46.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:46.112+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:46.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:46.042+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:46.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:46.042+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:46.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:46.042+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:46.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:46.043+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:47.138 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:46.978+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:47.139 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:46.978+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:47.139 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:46.978+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:47.139 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:46.979+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:47.139 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:47.061+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:47.139 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:47.061+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:47.139 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:47.061+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:47.139 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:47.062+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:47.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:47.139+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:47.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:47.139+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:47.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:47.139+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:47.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:47.140+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:48.282 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:47.930+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:48.282 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:47.930+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:48.282 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:47.930+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:48.282 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:47.930+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:48.283 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:48.146+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:48.283 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:48.146+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:48.283 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:48.146+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:48.283 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:48.147+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:48.283 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:48.107+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:48.283 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:48.107+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:48.283 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:48.107+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:48.284 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:48.107+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:48.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:23:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:23:48.647+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11416 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:23:49.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:48.945+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:49.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:48.945+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:49.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:48.945+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:49.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:48.946+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:49.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:49.157+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:49.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:49.157+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:49.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:49.157+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:49.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:49.158+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:49.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:49.140+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:49.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:49.140+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:49.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:49.140+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:49.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:49.141+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:50.095 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:49.985+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:50.095 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:49.985+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:50.095 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:49.985+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:50.095 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:49.986+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:50.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:50.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:50.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:50.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:50.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:50.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:50.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:50.118+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:50.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:50.131+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:50.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:50.131+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:50.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:50.131+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:50.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:50.131+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:51.093 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:51.008+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:51.093 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:51.008+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:51.094 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:51.008+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:51.094 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:51.008+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:51.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:51.118+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:51.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:51.118+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:51.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:51.118+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:51.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:51.119+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:51.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:51.095+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:51.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:51.095+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:51.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:51.095+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:51.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:51.095+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:52.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:52.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:52.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:52.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:52.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:52.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:52.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:52.134+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:52.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:52.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:52.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:52.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:52.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:52.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:52.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:52.034+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:52.133+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:52.133+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:52.133+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:52.133+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:53.318 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:52.987+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:53.318 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:52.987+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:53.318 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:52.987+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:53.318 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:52.987+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:53.319 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:53.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:53.319 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:53.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:53.319 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:53.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:53.319 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:53.162+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:53.319 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:53.099+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:53.319 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:53.099+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:53.319 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:53.099+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:53.319 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:53.099+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:53.920 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:23:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:23:53.648+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11465 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:23:54.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:54.013+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:54.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:54.013+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:54.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:54.013+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:54.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:54.014+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:54.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:54.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:54.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:54.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:54.095+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:54.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:54.211+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:54.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:54.211+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:54.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:54.211+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:54.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:54.212+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:55.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:55.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:55.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:55.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:55.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:55.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:55.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:55.077+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:55.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:55.200+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:55.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:55.200+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:55.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:55.200+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:55.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:55.201+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:54.972+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:54.972+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:54.972+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:54.973+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:56.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:56.014+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:56.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:56.014+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:56.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:56.014+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:56.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:56.015+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:56.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:56.153+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:56.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:56.153+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:56.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:56.153+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:56.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:56.154+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:56.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:56.086+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:56.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:56.086+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:56.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:56.086+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:56.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:56.087+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:57.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:57.044+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:57.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:57.044+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:57.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:57.044+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:57.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:57.045+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:57.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:57.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:57.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:57.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:57.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:57.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:57.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:57.173+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:57.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:56.981+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:57.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:56.981+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:57.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:56.981+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:57.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:56.983+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:58.354 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:57.999+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:58.354 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:57.999+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:58.355 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:57.999+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:58.355 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:58.000+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:58.355 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:58.155+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:58.355 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:58.155+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:58.355 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:58.155+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:58.355 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:58.155+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:58.356 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:58.068+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:58.356 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:58.068+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:58.356 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:58.068+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:58.356 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:58.069+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:58.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:23:58.650+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11515 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:23:59.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:58.962+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:59.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:58.962+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:59.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:58.962+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:23:59.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:58.963+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:23:59.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:59.073+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:59.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:59.073+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:59.073+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:23:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:23:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:23:59.074+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:23:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:59.192+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:59.192+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:59.192+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:23:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:23:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:23:59.194+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:00.210 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:00.114+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:00.210 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:00.114+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:00.210 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:00.114+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:00.210 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:00.114+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:00.210 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:59.939+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:00.210 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:59.939+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:00.210 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:59.939+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:00.210 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:23:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:23:59.940+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:00.660 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:00.211+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:00.660 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:00.211+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:00.660 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:00.211+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:00.660 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:00.212+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-27T04:24:00.866+0000 7faa1e9ca700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T04:24:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-04-27T04:24:01.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:01.085+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:01.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:01.085+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:01.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:01.085+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:01.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:01.086+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:01.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:00.910+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:01.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:00.910+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:01.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:00.910+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:01.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:00.912+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:01.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:01.225+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:01.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:01.225+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:01.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:01.225+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:01.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:01.226+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:01.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T04:24:01.833 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephadm.py", line 1475, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -e sha1=4e213a58e787437ff508e6c890ccd5fd32dc96b5 -- bash -c 'ceph orch ps'" 2024-04-27T04:24:02.045 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ede868120d52474badaab00dc4a2e71a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephadm.py", line 1475, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -e sha1=4e213a58e787437ff508e6c890ccd5fd32dc96b5 -- bash -c 'ceph orch ps'" 2024-04-27T04:24:02.048 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-27T04:24:02.061 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-27T04:24:02.062 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T04:24:02.062 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T04:24:02.084 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T04:24:02.084 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T04:24:02.109 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:01.954+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:02.109 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:01.954+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:02.109 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:01.954+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:02.109 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:01.955+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:02.110 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:02.063+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:02.110 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:02.063+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:02.110 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:02.063+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:02.110 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:02.064+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:02.127 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T04:24:02.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:02.224+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:02.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:02.224+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:02.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:02.224+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:02.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:02.225+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:03.034 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:03.004+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:03.034 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:03.004+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:03.034 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:03.004+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:03.035 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:03.006+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:03.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:03.265+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:03.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:03.265+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:03.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:03.265+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:03.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:03.266+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:03.397 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:03.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:03.398 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:03.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:03.398 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:03.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:03.398 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:03.052+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:03.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:24:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:24:03.652+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11565 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:24:04.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:04.070+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:04.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:04.070+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:04.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:04.070+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:04.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:04.071+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:04.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:04.266+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:04.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:04.266+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:04.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:04.266+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:04.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:04.268+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:04.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:04.022+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:04.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:04.022+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:04.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:04.022+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:04.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:04.023+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:05.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:05.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:05.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:05.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:05.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:05.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:05.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:05.102+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:05.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:05.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:05.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:05.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:05.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:05.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:05.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:05.273+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:05.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:05.036+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:05.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:05.036+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:05.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:05.036+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:05.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:05.037+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:06.319 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:06.118+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:06.320 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:06.118+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:06.320 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:06.118+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:06.320 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:06.119+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:06.320 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:06.003+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:06.320 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:06.003+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:06.321 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:06.003+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:06.321 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:06.004+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:06.321 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:06.321+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:06.321 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:06.321+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:06.321 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:06.321+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:06.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:06.322+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:07.330 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:07.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:07.331 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:07.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:07.331 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:07.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:07.331 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:07.112+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:07.331 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:07.032+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:07.331 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:07.032+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:07.331 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:07.032+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:07.331 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:07.033+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:07.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:07.332+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:07.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:07.332+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:07.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:07.332+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:07.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:07.332+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:08.340 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:07.985+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:08.341 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:07.985+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:08.341 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:07.985+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:08.341 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:07.986+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:08.341 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:08.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:08.341 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:08.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:08.342 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:08.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:08.342 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:08.077+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:08.651 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:08.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:08.652 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:08.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:08.652 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:08.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:08.652 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:08.343+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:08.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:24:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:24:08.653+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11616 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:24:09.300 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:09.099+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:09.300 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:09.099+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:09.301 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:09.099+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:09.301 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:09.100+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:09.301 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:09.006+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:09.301 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:09.006+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:09.301 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:09.006+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:09.301 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:09.007+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:09.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:09.301+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:09.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:09.301+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:09.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:09.301+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:09.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:09.303+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:09.961+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:09.961+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:09.961+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:10.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:09.962+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:10.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:10.112+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:10.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:10.112+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:10.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:10.112+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:10.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:10.113+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:10.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:10.283+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:10.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:10.283+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:10.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:10.283+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:10.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:10.283+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:11.275 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:11.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:11.276 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:11.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:11.276 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:11.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:11.276 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:11.095+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:11.276 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:10.939+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:11.276 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:10.939+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:11.277 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:10.939+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:11.277 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:10.940+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:11.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:11.277+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:11.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:11.277+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:11.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:11.277+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:11.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:11.278+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:12.280 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:11.986+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:12.281 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:11.986+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:12.281 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:11.986+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:12.281 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:11.987+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:12.281 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:12.123+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:12.282 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:12.123+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:12.282 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:12.123+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:12.282 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:12.124+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:12.577 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:12.282+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:12.577 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:12.282+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:12.578 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:12.282+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:12.578 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:12.283+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:13.280 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:13.115+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:13.281 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:13.115+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:13.281 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:13.115+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:13.281 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:13.115+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:13.281 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:12.950+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:13.281 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:12.950+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:13.281 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:12.950+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:13.281 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:12.951+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:13.576 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:13.281+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:13.576 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:13.281+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:13.577 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:13.281+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:13.577 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:13.282+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:13.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:24:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:24:13.654+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11666 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:24:14.250 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:14.068+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:14.251 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:14.068+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:14.251 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:14.068+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:14.251 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:14.069+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:14.252 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:13.956+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:14.252 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:13.956+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:14.252 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:13.956+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:14.252 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:13.957+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:14.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:14.252+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:14.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:14.252+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:14.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:14.252+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:14.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:14.253+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:15.279 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:15.049+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:15.279 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:15.049+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:15.279 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:15.049+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:15.280 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:15.050+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:15.280 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:14.994+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:15.280 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:14.994+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:15.280 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:14.994+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:15.280 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:14.995+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:15.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:15.280+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:15.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:15.280+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:15.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:15.280+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:15.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:15.281+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:16.315 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:16.059+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:16.315 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:16.059+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:16.315 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:16.059+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:16.315 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:16.060+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:16.316 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:15.952+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:16.316 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:15.952+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:16.316 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:15.952+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:16.316 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:15.953+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:16.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:16.316+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:16.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:16.316+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:16.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:16.316+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:16.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:16.317+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:17.334 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:17.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:17.334 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:17.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:17.334 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:17.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:17.335 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:17.040+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:17.335 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:16.987+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:17.335 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:16.987+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:17.335 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:16.987+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:17.335 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:16.989+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:17.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:17.335+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:17.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:17.335+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:17.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:17.335+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:17.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:17.336+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:18.371 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:18.054+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:18.371 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:18.054+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:18.371 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:18.054+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:18.371 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:18.055+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:18.371 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:17.958+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:18.372 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:17.958+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:18.372 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:17.958+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:18.372 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:17.960+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:18.653 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:18.372+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:18.654 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:18.372+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:18.654 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:18.372+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:18.654 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:18.373+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:18.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:24:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:24:18.655+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11717 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:24:19.325 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:19.065+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:19.326 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:19.065+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:19.326 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:19.065+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:19.326 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:19.065+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:19.327 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:18.979+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:19.327 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:18.979+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:19.327 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:18.979+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:19.327 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:18.980+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:19.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:19.326+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:19.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:19.326+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:19.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:19.326+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:19.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:19.327+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:20.339 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:19.960+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:20.340 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:19.960+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:20.340 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:19.960+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:20.340 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:19.960+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:20.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:20.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:20.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:20.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:20.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:20.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:20.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:20.083+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:20.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:20.341+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:20.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:20.341+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:20.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:20.341+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:20.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:20.342+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:21.307 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:21.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:21.307 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:21.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:21.307 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:21.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:21.307 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:21.062+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:21.308 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:20.929+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:21.308 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:20.929+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:21.308 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:20.929+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:21.308 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:20.930+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:21.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:21.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:21.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:21.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:21.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:21.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:21.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:21.309+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:22.305 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:22.014+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:22.305 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:22.014+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:22.306 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:22.014+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:22.306 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:22.015+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:22.306 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:21.923+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:22.306 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:21.923+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:22.306 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:21.923+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:22.306 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:21.924+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:22.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:22.306+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:22.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:22.306+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:22.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:22.306+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:22.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:22.308+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:23.352 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:23.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:23.352 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:23.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:23.352 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:23.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:23.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:23.042+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:23.353 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:22.930+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:23.353 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:22.930+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:23.353 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:22.930+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:23.353 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:22.931+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:23.655 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:23.353+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:23.656 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:23.353+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:23.656 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:23.353+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:23.656 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:23.354+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:23.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:24:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:24:23.656+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11767 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:24:24.323 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:24.063+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:24.323 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:24.063+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:24.323 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:24.063+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:24.324 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:24.064+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:24.324 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:23.927+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:24.324 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:23.927+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:24.324 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:23.927+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:24.324 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:23.929+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:24.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:24.324+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:24.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:24.324+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:24.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:24.324+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:24.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:24.325+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:25.235 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:24.932+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:25.236 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:24.932+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:25.236 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:24.932+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:25.236 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:24.933+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:25.236 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:25.090+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:25.236 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:25.090+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:25.237 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:25.090+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:25.237 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:25.092+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:25.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:25.339+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:25.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:25.339+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:25.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:25.339+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:25.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:25.339+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:26.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:26.081+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:26.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:26.081+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:26.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:26.081+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:26.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:26.082+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:26.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:25.920+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:26.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:25.920+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:26.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:25.920+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:26.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:25.921+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:26.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:26.305+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:26.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:26.305+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:26.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:26.305+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:26.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:26.306+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:27.332 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:27.106+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:27.332 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:27.106+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:27.332 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:27.106+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:27.333 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:27.107+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:27.333 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:26.950+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:27.333 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:26.950+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:27.333 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:26.950+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:27.333 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:26.951+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:27.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:27.333+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:27.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:27.333+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:27.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:27.333+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:27.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:27.334+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:28.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:28.112+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:28.369 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:28.112+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:28.369 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:28.112+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:28.369 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:28.113+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:28.369 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:27.928+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:28.369 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:27.928+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:28.369 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:27.928+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:28.370 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:27.929+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:28.657 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:28.370+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:28.657 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:28.370+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:28.658 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:28.370+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:28.658 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:28.371+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:28.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:24:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:24:28.658+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11818 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:24:28.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:28.907+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:28.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:28.907+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:28.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:28.907+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:28.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:28.908+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:29.410 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:29.151+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:29.410 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:29.151+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:29.410 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:29.151+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:29.410 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:29.152+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:29.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:29.411+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:29.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:29.411+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:29.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:29.411+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:29.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:29.412+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:30.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:29.909+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:30.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:29.909+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:30.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:29.909+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:30.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:29.910+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:30.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:30.132+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:30.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:30.132+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:30.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:30.132+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:30.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:30.133+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:30.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:30.386+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:30.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:30.386+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:30.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:30.386+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:30.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:30.386+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:31.380 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:31.122+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:31.380 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:31.122+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:31.380 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:31.122+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:31.381 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:31.122+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:31.381 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:30.926+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:31.381 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:30.926+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:31.381 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:30.926+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:31.381 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:30.926+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:31.649 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:31.411+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:31.649 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:31.411+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:31.649 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:31.411+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:31.649 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:31.412+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:31.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:31.897+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:31.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:31.897+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:31.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:31.897+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:31.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:31.898+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:32.396 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:32.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:32.396 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:32.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:32.396 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:32.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:32.396 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:32.093+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:32.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:32.397+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:32.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:32.397+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:32.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:32.397+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:32.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:32.398+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:33.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:33.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:33.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:33.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:33.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:33.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:33.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:33.102+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:33.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:32.930+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:33.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:32.930+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:33.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:32.930+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:33.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:32.931+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:33.671 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:24:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:24:33.659+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11868 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:24:33.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:33.419+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:33.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:33.419+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:33.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:33.419+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:33.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:33.420+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:34.397 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:34.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:34.398 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:34.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:34.398 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:34.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:34.398 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:34.078+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:34.398 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:33.943+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:34.398 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:33.943+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:34.399 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:33.943+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:34.399 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:33.944+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:34.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:34.399+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:34.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:34.399+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:34.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:34.399+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:34.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:34.399+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:35.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:35.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:35.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:35.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:35.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:35.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:35.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:35.046+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:35.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:34.899+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:35.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:34.899+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:35.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:34.899+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:35.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:34.900+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:35.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:35.378+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:35.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:35.378+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:35.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:35.378+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:35.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:35.379+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:36.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:36.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:36.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:36.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:36.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:36.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:36.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:36.084+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:36.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:35.921+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:36.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:35.921+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:36.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:35.921+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:36.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:35.922+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:36.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:36.379+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:36.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:36.379+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:36.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:36.379+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:36.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:36.379+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:37.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:37.079+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:37.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:37.079+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:37.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:37.079+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:37.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:37.080+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:37.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:36.877+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:37.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:36.877+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:37.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:36.877+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:37.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:36.877+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:37.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:37.343+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:37.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:37.343+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:37.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:37.343+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:37.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:37.343+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:38.082 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:37.878+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:38.082 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:37.878+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:38.082 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:37.878+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:38.082 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:37.879+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:38.332 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:38.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:38.332 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:38.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:38.332 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:38.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:38.333 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:38.084+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:38.659 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:38.333+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:38.659 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:38.333+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:38.660 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:38.333+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:38.660 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:38.334+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:38.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:24:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:24:38.660+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11919 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:24:38.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:38.886+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:38.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:38.886+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:38.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:38.886+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:38.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:38.887+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:39.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:39.086+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:39.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:39.086+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:39.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:39.086+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:39.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:39.087+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:39.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:39.287+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:39.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:39.287+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:39.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:39.287+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:39.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:39.287+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:40.259 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:39.933+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:40.260 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:39.933+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:40.260 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:39.933+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:40.260 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:39.933+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:40.260 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:40.043+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:40.260 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:40.043+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:40.260 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:40.043+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:40.260 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:40.044+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:40.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:40.260+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:40.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:40.260+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:40.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:40.260+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:40.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:40.261+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:41.298 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:41.005+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:41.298 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:41.005+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:41.299 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:41.005+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:41.299 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:41.006+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:41.299 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:40.952+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:41.299 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:40.952+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:41.299 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:40.952+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:41.299 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:40.952+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:41.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:41.299+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:41.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:41.299+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:41.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:41.299+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:41.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:41.300+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:42.339 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:41.999+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:42.339 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:41.999+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:42.339 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:41.999+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:42.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:42.001+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:42.340 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:41.981+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:42.340 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:41.981+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:42.340 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:41.981+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:42.340 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:41.982+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:42.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:42.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:42.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:42.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:42.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:42.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:42.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:42.341+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:43.323 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:43.026+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:43.323 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:43.026+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:43.324 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:43.026+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:43.324 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:43.026+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:43.324 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:43.019+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:43.324 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:43.019+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:43.324 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:43.019+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:43.324 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:43.020+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:43.597 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:43.324+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:43.597 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:43.324+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:43.597 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:43.324+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:43.597 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:43.325+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:43.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:24:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:24:43.661+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 11969 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:24:44.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:44.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:44.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:44.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:44.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:44.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:44.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:44.039+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:44.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:44.276+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:44.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:44.276+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:44.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:44.276+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:44.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:44.276+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:44.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:44.064+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:44.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:44.064+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:44.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:44.064+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:44.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:44.065+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:45.244+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:45.244+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:45.244+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:45.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:45.246+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:45.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:45.024+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:45.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:45.024+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:45.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:45.024+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:45.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:45.024+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:45.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:45.013+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:45.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:45.013+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:45.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:45.013+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:45.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:45.014+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:46.269 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:46.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:46.269 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:46.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:46.269 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:46.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:46.269 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:46.051+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:46.270 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:45.983+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:46.270 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:45.983+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:46.270 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:45.983+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:46.270 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:45.983+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:46.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:46.270+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:46.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:46.270+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:46.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:46.270+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:46.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:46.272+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:47.297 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:47.009+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:47.297 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:47.009+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:47.297 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:47.009+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:47.297 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:47.010+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:47.298 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:46.953+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:47.298 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:46.953+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:47.298 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:46.953+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:47.298 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:46.954+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:47.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:47.298+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:47.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:47.298+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:47.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:47.298+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:47.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:47.299+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:48.267 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:47.948+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:48.268 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:47.948+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:48.268 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:47.948+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:48.268 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:47.949+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:48.268 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:47.993+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:48.268 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:47.993+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:48.269 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:47.993+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:48.269 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:47.994+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:48.661 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:48.269+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:48.662 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:48.269+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:48.662 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:48.269+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:48.662 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:48.270+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:48.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:24:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:24:48.663+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12020 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:24:48.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:48.902+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:48.921 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:48.902+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:48.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:48.902+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:48.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:48.902+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:49.292 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:49.037+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:49.293 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:49.037+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:49.293 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:49.037+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:49.293 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:49.039+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:49.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:49.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:49.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:49.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:49.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:49.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:49.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:49.295+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:50.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:49.915+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:50.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:49.915+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:50.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:49.915+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:50.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:49.916+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:50.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:50.042+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:50.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:50.042+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:50.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:50.042+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:50.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:50.043+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:50.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:50.330+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:50.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:50.330+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:50.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:50.330+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:50.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:50.331+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:51.376 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:51.005+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:51.377 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:51.005+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:51.377 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:51.005+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:51.377 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:51.006+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:51.378 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:50.944+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:51.378 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:50.944+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:51.378 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:50.944+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:51.378 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:50.945+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:51.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:51.377+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:51.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:51.377+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:51.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:51.377+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:51.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:51.378+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:52.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:52.037+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:52.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:52.037+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:52.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:52.037+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:52.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:52.038+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:52.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:51.904+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:52.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:51.904+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:52.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:51.904+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:52.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:51.904+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:52.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:52.365+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:52.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:52.365+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:52.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:52.365+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:52.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:52.366+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:53.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:53.034+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:53.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:53.034+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:53.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:53.034+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:53.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:53.034+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:53.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:52.885+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:53.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:52.885+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:53.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:52.885+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:53.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:52.886+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:53.663 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:53.351+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:53.663 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:53.351+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:53.663 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:53.351+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:53.663 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:53.352+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:53.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:24:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:24:53.664+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12069 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:24:53.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:53.839+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:53.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:53.839+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:53.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:53.839+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:53.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:53.840+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:54.359 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:54.018+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:54.359 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:54.018+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:54.359 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:54.018+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:54.359 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:54.019+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:54.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:54.360+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:54.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:54.360+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:54.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:54.360+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:54.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:54.361+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:55.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:54.976+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:55.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:54.976+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:55.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:54.976+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:55.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:54.977+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:55.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:54.801+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:55.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:54.801+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:55.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:54.801+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.816403+0000 (oldest deadline 2024-04-27T04:14:45.816403+0000) 2024-04-27T04:24:55.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:54.802+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:55.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:55.328+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:55.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:55.328+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:55.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:55.328+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:55.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:55.329+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:56.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:55.811+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:56.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:56.026+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:56.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:56.026+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:56.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:56.026+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:56.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:56.027+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:56.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:56.318+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:56.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:56.318+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:56.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:56.318+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:56.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:56.319+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:57.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:57.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:57.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:57.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:57.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:57.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:57.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:57.041+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:57.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:56.846+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:57.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:57.278+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:57.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:57.278+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:57.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:57.278+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:57.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:57.279+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:58.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:57.886+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:58.171 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:58.069+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:58.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:58.069+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:58.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:58.069+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:58.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:58.070+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:58.664 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:58.260+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:58.664 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:58.260+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:58.664 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:58.260+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:58.664 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:58.261+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:58.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:24:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:24:58.665+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12119 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:24:58.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:58.845+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:24:59.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:59.222+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:59.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:59.222+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:59.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:59.222+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:25.546916+0000 (oldest deadline 2024-04-27T04:14:45.546916+0000) 2024-04-27T04:24:59.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:24:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:24:59.222+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:24:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:59.105+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:59.105+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:59.105+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:24:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:24:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:24:59.106+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:00.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:24:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:24:59.884+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:00.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:00.088+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:25:00.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:00.088+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:25:00.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:00.088+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:25:00.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:00.089+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:00.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:00.230+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:01.127 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:00.859+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:01.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:01.266+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:01.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:01.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:25:01.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:01.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:25:01.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:01.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:25:01.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:01.129+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:02.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:01.901+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:02.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:02.225+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:02.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:02.176+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:25:02.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:02.176+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:25:02.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:02.176+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:14:29.375210+0000 (oldest deadline 2024-04-27T04:14:49.375210+0000) 2024-04-27T04:25:02.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:02.177+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:03.208 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:02.936+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:03.598 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:03.209+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:03.599 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:03.211+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:03.886 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:25:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:25:03.666+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12167 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:25:04.149 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:03.960+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:04.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:04.246+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:04.683 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:04.219+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:05.241 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:04.985+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:05.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:05.242+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:05.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:05.282+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:06.268 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:05.962+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:06.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:06.270+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:06.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:06.280+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:07.246 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:06.932+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:07.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:07.318+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:07.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:07.247+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:08.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:07.918+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:08.666 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:08.356+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:08.667 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:08.245+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:08.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:25:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:25:08.668+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12217 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:25:09.223 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:08.952+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:09.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:09.383+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:09.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:09.224+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:10.371 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:10.227+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:10.372 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:09.994+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:10.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:10.372+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:11.316 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:11.023+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:11.317 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:11.253+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:11.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:11.404+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:12.380 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:12.023+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:12.381 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:12.265+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:12.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:12.381+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:13.296 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:13.011+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:13.600 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:13.421+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:13.600 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:13.296+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:13.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:25:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:25:13.668+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12265 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:25:14.256 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:14.011+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:14.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:14.462+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:14.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:14.256+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:15.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:15.061+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:15.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:15.261+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:15.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:15.454+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:16.284 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:16.017+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:16.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:16.284+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:16.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:16.464+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:17.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:17.060+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:17.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:17.258+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:17.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:17.500+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:18.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:18.042+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:18.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:18.213+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:18.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:25:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:25:18.669+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12315 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:25:18.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:18.460+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:19.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:19.175+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:19.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:19.010+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:19.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:19.010+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:19.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:19.010+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:19.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:19.011+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:19.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:19.490+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:20.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:20.135+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:20.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:20.059+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:20.059+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:20.059+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:20.060+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:20.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:20.497+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:21.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:21.165+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:21.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:21.021+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:21.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:21.021+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:21.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:21.021+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:21.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:21.022+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:21.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:21.492+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:22.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:22.150+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:22.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:21.988+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:21.988+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:21.988+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:21.988+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:22.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:22.478+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:23.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:23.180+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:23.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:23.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:23.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:23.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:23.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:23.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:23.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:23.005+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:23.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:25:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:25:23.670+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12364 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:25:23.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:23.460+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:24.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:24.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:24.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:24.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:24.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:24.001+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:24.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:24.218+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:24.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:24.218+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:24.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:24.218+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:24.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:24.218+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:24.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:24.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:24.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:24.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:24.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:24.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:24.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:24.502+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:25.238 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:24.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:25.239 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:24.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:25.239 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:24.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:25.239 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:24.978+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:25.532 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:25.239+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:25.532 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:25.239+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:25.532 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:25.239+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:25.532 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:25.240+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:25.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:25.532+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:25.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:25.532+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:25.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:25.532+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:25.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:25.533+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:26.223 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:25.955+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:26.223 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:25.955+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:26.224 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:25.955+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:26.224 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:25.956+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:26.499 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:26.224+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:26.500 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:26.224+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:26.500 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:26.224+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:26.500 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:26.225+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:26.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:26.500+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:26.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:26.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:26.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:26.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:26.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:26.501+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:27.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:26.965+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:27.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:26.965+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:27.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:26.965+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:27.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:26.967+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:27.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:27.210+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:27.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:27.210+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:27.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:27.210+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:27.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:27.211+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:27.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:27.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:27.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:27.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:27.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:27.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:27.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:27.461+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:28.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:27.983+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:28.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:27.983+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:28.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:27.983+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:28.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:27.984+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:28.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:28.209+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:28.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:28.209+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:28.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:28.209+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:28.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:28.210+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:28.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:25:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:25:28.672+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12414 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:25:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:28.478+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:28.478+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:28.478+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:28.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:28.479+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:29.187 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:28.936+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:29.188 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:28.936+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:29.188 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:28.936+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:29.188 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:28.937+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:29.472 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:29.188+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:29.472 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:29.188+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:29.472 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:29.188+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:29.472 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:29.189+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:29.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:29.473+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:29.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:29.473+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:29.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:29.473+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:29.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:29.474+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:30.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:30.191+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:30.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:30.191+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:30.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:30.191+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:30.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:30.192+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:30.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:29.943+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:30.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:29.943+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:30.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:29.943+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:30.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:29.945+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:30.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:30.519+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:30.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:30.519+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:30.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:30.519+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:30.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:30.519+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:31.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:30.990+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:31.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:30.990+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:31.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:30.990+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:31.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:30.991+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:31.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:31.192+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:31.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:31.192+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:31.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:31.192+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:31.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:31.193+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:31.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:31.526+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:31.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:31.526+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:31.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:31.526+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:31.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:31.527+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:32.224 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:31.974+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:32.224 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:31.974+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:32.225 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:31.974+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:32.225 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:31.974+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:32.514 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:32.225+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:32.515 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:32.225+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:32.515 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:32.225+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:32.515 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:32.226+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:32.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:32.515+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:32.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:32.515+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:32.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:32.515+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:32.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:32.516+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:33.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:32.937+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:33.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:32.937+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:33.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:32.937+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:33.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:32.938+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:33.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:33.178+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:33.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:33.178+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:33.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:33.178+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:33.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:33.178+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:33.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:25:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:25:33.674+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12463 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:25:33.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:33.484+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:33.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:33.484+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:33.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:33.484+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:33.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:33.485+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:34.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:34.136+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:34.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:34.136+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:34.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:34.136+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:34.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:34.137+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:34.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:33.932+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:34.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:33.932+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:34.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:33.932+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:34.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:33.933+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:34.895 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:34.449+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:34.895 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:34.449+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:34.895 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:34.449+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:34.895 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:34.449+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:35.165 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:34.895+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:35.165 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:34.895+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:35.165 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:34.895+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:35.166 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:34.897+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:35.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:35.166+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:35.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:35.166+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:35.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:35.166+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:35.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:35.167+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:35.868 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:35.444+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:35.868 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:35.444+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:35.868 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:35.444+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:35.868 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:35.445+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:36.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:35.869+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:36.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:35.869+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:36.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:35.869+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:36.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:35.869+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:36.473 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:36.214+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:36.474 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:36.214+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:36.474 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:36.214+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:36.474 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:36.215+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:36.833 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:36.474+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:36.834 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:36.474+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:36.834 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:36.474+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:36.834 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:36.475+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:37.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:36.834+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:37.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:36.834+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:37.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:36.834+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:37.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:36.835+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:37.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:37.235+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:37.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:37.235+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:37.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:37.235+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:37.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:37.236+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:37.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:37.469+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:37.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:37.469+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:37.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:37.469+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:37.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:37.470+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:38.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:37.828+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:38.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:37.828+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:38.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:37.828+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:38.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:37.829+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:38.495 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:38.214+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:38.496 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:38.214+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:38.496 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:38.214+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:38.496 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:38.215+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:38.864 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:25:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:25:38.675+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12514 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:25:38.864 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:38.496+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:38.864 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:38.496+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:38.864 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:38.496+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:38.865 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:38.497+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:39.150 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:38.865+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:39.150 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:38.865+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:39.150 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:38.865+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:39.150 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:38.865+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:39.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:39.241+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:39.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:39.241+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:39.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:39.241+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:39.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:39.242+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:39.885 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:39.511+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:39.886 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:39.511+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:39.886 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:39.511+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:39.886 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:39.513+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:40.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:39.886+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:40.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:39.886+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:40.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:39.886+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:40.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:39.887+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:40.507 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:40.206+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:40.508 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:40.206+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:40.508 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:40.206+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:40.508 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:40.207+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:40.841 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:40.508+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:40.841 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:40.508+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:40.841 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:40.508+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:40.841 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:40.509+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:41.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:40.842+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:41.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:40.842+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:41.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:40.842+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:41.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:40.843+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:41.495 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:41.200+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:41.496 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:41.200+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:41.496 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:41.200+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:41.496 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:41.200+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:41.793 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:41.496+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:41.793 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:41.496+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:41.793 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:41.496+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:41.793 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:41.497+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:42.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:41.794+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:42.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:41.794+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:42.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:41.794+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:42.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:41.795+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:42.475 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:42.180+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:42.475 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:42.180+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:42.475 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:42.180+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:42.476 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:42.181+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:42.825 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:42.476+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:42.826 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:42.476+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:42.826 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:42.476+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:42.826 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:42.477+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:43.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:42.826+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:43.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:42.826+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:43.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:42.826+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:43.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:42.827+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:43.473 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:43.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:43.473 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:43.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:43.474 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:43.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:43.474 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:43.177+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:43.799 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:25:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:25:43.676+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12564 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:25:43.799 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:43.474+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:43.800 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:43.474+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:43.800 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:43.474+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:43.800 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:43.475+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:44.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:43.800+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:44.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:43.800+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:44.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:43.800+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:44.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:43.801+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:44.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:44.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:44.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:44.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:44.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:44.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:44.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:44.173+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:44.798 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:44.502+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:44.798 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:44.502+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:44.798 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:44.502+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:44.798 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:44.503+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:45.108 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:44.799+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:45.108 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:44.799+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:45.109 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:44.799+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:45.109 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:44.800+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:45.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:45.163+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:45.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:45.163+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:45.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:45.163+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:45.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:45.164+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:45.762 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:45.481+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:45.762 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:45.481+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:45.762 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:45.481+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:45.762 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:45.482+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:46.126 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:45.762+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:46.126 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:45.762+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:46.126 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:45.762+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:46.126 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:45.764+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:46.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:46.126+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:46.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:46.126+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:46.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:46.126+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:46.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:46.127+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:46.768 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:46.452+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:46.769 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:46.452+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:46.769 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:46.452+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:46.769 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:46.454+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:47.140 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:46.769+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:47.140 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:46.769+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:47.140 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:46.769+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:47.140 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:46.770+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:47.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:47.140+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:47.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:47.140+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:47.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:47.140+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:47.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:47.141+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:47.800 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:47.500+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:47.800 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:47.500+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:47.800 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:47.500+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:47.800 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:47.501+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:48.156 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:47.800+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:48.157 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:47.800+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:48.157 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:47.800+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:48.157 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:47.802+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:48.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:48.157+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:48.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:48.157+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:48.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:48.157+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:48.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:48.158+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:48.844 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:25:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:25:48.677+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12617 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:25:48.844 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:48.495+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:48.844 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:48.495+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:48.844 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:48.495+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:48.845 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:48.496+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:49.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:48.844+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:49.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:48.844+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:49.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:48.844+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:49.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:48.846+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:49.515 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:49.182+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:49.515 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:49.182+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:49.515 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:49.182+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:49.515 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:49.184+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:49.859 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:49.515+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:49.859 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:49.515+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:49.859 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:49.515+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:49.859 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:49.516+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:50.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:49.860+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:50.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:49.860+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:50.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:49.860+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:50.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:49.861+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:50.550 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:50.186+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:50.550 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:50.186+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:50.551 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:50.186+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:50.551 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:50.187+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:50.865 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:50.551+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:50.865 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:50.551+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:50.866 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:50.551+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:50.866 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:50.552+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:51.150 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:50.866+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:51.151 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:50.866+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:51.151 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:50.866+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:51.151 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:50.867+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:51.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:51.151+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:51.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:51.151+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:51.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:51.151+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:51.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:51.152+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:51.871 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:51.548+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:51.872 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:51.548+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:51.872 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:51.548+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:51.872 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:51.548+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:52.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:51.872+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:52.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:51.872+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:52.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:51.872+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:52.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:51.874+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:52.582 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:52.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:52.582 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:52.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:52.582 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:52.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:52.583 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:52.173+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:52.853 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:52.583+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:52.854 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:52.583+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:52.854 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:52.583+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:52.854 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:52.584+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:53.127 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:52.854+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:53.127 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:52.854+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:53.128 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:52.854+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:53.128 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:52.856+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:53.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:53.128+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:53.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:53.128+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:53.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:53.128+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:53.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:53.128+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:53.901 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:25:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:25:53.679+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12668 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:25:53.902 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:53.570+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:53.902 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:53.570+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:53.902 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:53.570+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:53.902 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:53.571+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:54.166 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:53.902+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:54.167 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:53.902+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:54.167 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:53.902+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:54.167 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:53.903+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:54.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:54.167+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:54.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:54.167+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:54.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:54.167+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:54.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:54.168+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:54.917 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:54.591+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:54.917 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:54.591+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:54.917 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:54.591+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:54.917 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:54.592+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:55.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:55.120+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:55.120+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:55.120+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:55.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:55.122+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:55.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:54.917+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:55.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:54.917+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:55.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:54.917+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:55.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:54.919+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:55.919 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:55.584+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:55.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:55.584+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:55.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:55.584+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:55.920 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:55.585+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:56.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:56.148+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:56.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:56.148+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:56.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:56.148+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:56.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:56.148+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:56.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:55.920+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:56.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:55.920+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:56.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:55.920+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:56.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:55.921+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:56.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:56.608+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:56.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:56.608+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:56.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:56.608+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:56.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:56.609+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:57.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:57.101+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:57.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:57.101+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:57.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:57.101+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:57.397 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:57.102+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:57.398 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:56.955+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:57.398 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:56.955+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:57.398 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:56.955+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:57.398 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:56.956+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:57.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:57.606+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:57.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:57.606+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:57.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:57.606+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:57.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:57.607+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:58.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:58.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:58.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:58.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:58.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:58.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:58.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:58.064+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:58.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:57.961+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:58.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:57.961+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:58.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:57.961+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:58.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:57.962+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:58.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:25:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:25:58.680+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12721 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:25:58.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:58.646+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:58.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:58.646+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:58.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:58.646+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:58.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:58.647+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:59.295 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:58.970+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:59.295 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:58.970+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:59.295 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:58.970+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:25:59.295 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:58.971+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:25:59.295 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:59.106+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:59.295 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:59.106+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:59.296 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:59.106+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:25:59.296 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:25:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:25:59.106+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:25:59.989 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:59.690+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:59.989 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:59.690+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:59.989 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:59.690+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:25:59.990 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:25:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:25:59.691+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:00.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:00.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:00.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:00.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:00.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:00.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:00.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:00.071+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:00.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:59.990+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:00.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:59.990+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:00.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:59.990+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:00.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:25:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:25:59.991+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:00.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:00.648+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:00.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:00.648+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:00.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:00.648+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:00.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:00.648+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:01.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:00.956+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:01.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:00.956+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:01.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:00.956+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:01.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:00.958+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:01.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:01.043+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:01.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:01.043+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:01.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:01.043+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:01.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:01.043+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:01.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:01.667+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:01.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:01.667+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:01.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:01.667+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:01.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:01.668+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:02.200 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-27T04:26:02.200 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-04-27T04:26:02.201 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T04:26:02.201 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T04:26:02.218 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-27T04:26:02.218 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T04:26:02.275 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T04:26:02.337 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:02.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:02.337 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:02.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:02.337 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:02.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:02.337 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:02.052+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:02.338 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:01.971+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:02.338 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:01.971+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:02.338 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:01.971+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:02.338 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:01.973+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:02.960 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:02.696+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:02.961 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:02.696+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:02.961 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:02.696+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:02.961 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:02.697+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:02.961 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:02.961+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:02.961 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:02.961+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:02.962 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:02.961+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:03.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:02.962+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:03.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:03.080+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:03.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:03.080+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:03.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:03.080+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:03.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:03.081+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:03.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:26:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:26:03.681+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12772 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:26:03.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:03.711+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:03.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:03.711+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:03.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:03.711+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:03.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:03.711+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:04.334 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:03.959+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:04.334 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:03.959+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:04.334 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:03.959+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:04.335 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:03.959+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:04.335 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:04.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:04.335 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:04.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:04.335 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:04.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:04.335 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:04.072+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:04.955 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:04.684+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:04.956 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:04.684+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:04.956 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:04.684+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:04.956 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:04.685+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:05.285 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:05.087+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:05.285 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:05.087+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:05.285 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:05.087+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:05.285 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:05.088+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:05.286 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:04.956+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:05.286 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:04.956+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:05.286 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:04.956+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:05.286 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:04.957+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:05.983 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:05.680+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:05.983 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:05.680+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:05.983 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:05.680+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:05.983 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:05.681+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:06.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:06.054+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:06.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:06.054+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:06.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:06.054+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:06.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:06.055+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:06.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:05.983+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:06.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:05.983+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:06.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:05.983+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:06.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:05.984+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:06.978 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:06.706+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:06.979 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:06.706+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:06.979 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:06.706+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:06.979 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:06.708+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:07.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:07.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:07.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:07.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:07.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:07.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:07.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:07.059+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:07.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:06.979+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:07.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:06.979+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:07.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:06.979+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:07.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:06.980+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:08.009 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:07.693+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:08.009 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:07.693+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:08.009 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:07.693+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:08.010 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:07.694+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:08.421 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:08.073+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:08.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:08.073+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:08.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:08.073+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:08.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:08.074+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:08.010+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:08.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:08.010+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:08.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:08.010+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:08.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:08.011+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:09.046 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:26:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:26:08.682+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12825 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:26:09.047 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:08.699+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:09.047 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:08.699+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:09.047 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:08.699+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:09.047 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:08.700+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:09.047 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:09.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:09.048 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:09.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:09.048 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:09.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:09.385 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:09.048+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:09.386 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:09.060+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:09.386 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:09.060+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:09.386 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:09.060+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:09.386 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:09.060+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:10.016 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:09.731+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:10.017 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:09.731+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:10.017 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:09.731+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:10.017 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:09.732+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:10.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:10.110+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:10.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:10.110+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:10.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:10.110+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:10.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:10.112+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:10.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:10.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:10.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:10.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:10.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:10.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:10.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:10.018+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:11.034 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:10.696+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:11.034 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:10.696+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:11.035 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:10.696+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:11.035 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:10.697+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:11.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:11.161+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:11.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:11.161+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:11.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:11.161+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:11.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:11.162+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:11.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:11.034+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:11.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:11.034+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:11.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:11.034+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:11.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:11.035+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:11.921 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:11.745+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:11.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:11.745+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:11.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:11.745+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:11.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:11.746+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:12.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:12.166+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:12.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:12.166+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:12.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:12.166+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:12.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:12.168+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:12.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:12.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:12.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:12.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:12.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:12.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:12.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:12.075+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:13.025 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:12.699+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:13.026 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:12.699+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:13.026 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:12.699+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:13.026 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:12.700+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:13.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:13.140+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:13.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:13.140+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:13.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:13.140+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:13.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:13.141+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:13.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:13.026+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:13.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:13.026+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:13.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:13.026+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:13.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:13.027+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:13.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:26:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:26:13.684+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12876 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:26:13.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:13.701+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:13.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:13.701+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:13.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:13.701+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:13.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:13.702+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:14.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:14.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:14.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:14.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:14.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:14.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:14.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:14.024+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:14.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:14.113+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:14.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:14.113+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:14.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:14.113+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:14.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:14.113+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:15.008 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:14.722+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:15.008 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:14.722+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:15.008 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:14.722+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:15.009 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:14.722+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:15.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:15.106+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:15.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:15.106+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:15.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:15.106+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:15.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:15.107+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:15.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:15.009+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:15.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:15.009+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:15.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:15.009+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:15.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:15.010+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:16.094 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:15.747+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:16.094 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:15.747+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:16.094 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:15.747+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:16.094 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:15.748+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:16.095 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:15.994+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:16.095 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:15.994+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:16.095 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:15.994+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:16.095 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:15.995+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:16.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:16.094+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:16.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:16.094+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:16.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:16.094+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:16.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:16.095+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:16.978 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:16.719+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:16.979 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:16.719+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:16.979 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:16.719+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:16.979 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:16.720+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:17.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:16.979+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:17.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:16.979+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:17.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:16.979+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:17.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:16.980+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:17.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:17.131+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:17.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:17.131+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:17.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:17.131+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:17.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:17.133+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:18.013 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:17.707+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:18.013 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:17.707+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:18.014 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:17.707+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:18.014 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:17.708+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:18.127+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:18.127+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:18.127+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:18.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:18.128+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:18.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:18.014+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:18.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:18.014+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:18.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:18.014+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:18.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:18.015+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:19.021 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:26:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:26:18.686+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12927 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:26:19.022 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:18.752+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:19.022 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:18.752+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:19.022 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:18.752+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:19.022 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:18.753+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:19.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:19.176+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:19.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:19.176+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:19.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:19.176+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:19.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:19.176+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:19.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:19.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:19.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:19.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:19.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:19.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:19.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:19.023+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:20.049 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:19.760+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:20.049 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:19.760+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:20.049 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:19.760+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:20.050 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:19.761+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:20.421 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:20.216+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:20.216+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:20.216+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:20.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:20.217+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:20.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:20.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:20.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:20.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:20.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:20.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:20.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:20.050+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:21.073 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:20.781+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:21.073 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:20.781+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:21.073 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:20.781+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:21.073 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:20.782+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:21.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:21.073+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:21.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:21.073+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:21.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:21.073+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:21.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:21.074+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:21.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:21.198+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:21.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:21.198+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:21.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:21.198+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:21.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:21.199+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:21.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:21.776+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:21.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:21.776+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:21.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:21.776+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:21.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:21.777+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:22.221+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:22.221+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:22.221+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:22.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:22.221+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:22.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:22.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:22.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:22.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:22.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:22.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:22.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:22.034+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:23.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:23.045+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:23.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:23.045+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:23.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:23.045+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:23.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:23.046+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:23.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:22.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:23.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:22.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:23.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:22.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:23.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:22.824+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:23.605 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:23.232+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:23.606 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:23.232+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:23.606 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:23.232+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:23.606 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:23.233+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:23.921 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:26:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:26:23.687+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 12979 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:26:23.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:23.802+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:23.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:23.802+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:23.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:23.802+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:23.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:23.803+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:24.249+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:24.249+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:24.249+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:24.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:24.250+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:24.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:24.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:24.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:24.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:24.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:24.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:24.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:24.036+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:25.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:24.769+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:25.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:24.769+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:25.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:24.769+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:25.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:24.770+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:25.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:24.998+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:25.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:24.998+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:25.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:24.998+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:25.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:24.999+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:25.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:25.292+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:25.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:25.292+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:25.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:25.292+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:25.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:25.293+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:26.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:25.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:26.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:25.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:26.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:25.962+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:26.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:25.963+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:26.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:25.759+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:26.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:25.759+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:26.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:25.759+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:26.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:25.760+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:26.336+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:26.336+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:26.336+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:26.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:26.337+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:26.994 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:26.729+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:26.994 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:26.729+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:26.994 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:26.729+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:26.994 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:26.730+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:27.360 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:26.994+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:27.360 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:26.994+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:27.361 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:26.994+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:27.361 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:26.995+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:27.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:27.361+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:27.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:27.361+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:27.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:27.361+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:27.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:27.362+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:28.117 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:27.778+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:28.117 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:27.778+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:28.117 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:27.778+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:28.118 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:27.780+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:28.118 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:27.965+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:28.118 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:27.965+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:28.118 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:27.965+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:28.118 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:27.966+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:28.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:28.342+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:28.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:28.342+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:28.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:28.342+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:28.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:28.342+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:29.171 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:26:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:26:28.688+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13032 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:26:29.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:28.936+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:29.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:28.936+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:29.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:28.936+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:29.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:28.937+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:29.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:28.800+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:29.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:28.800+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:29.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:28.800+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:29.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:28.801+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:29.298+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:29.298+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:29.298+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:29.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:29.299+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:30.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:29.771+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:30.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:29.771+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:30.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:29.771+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:30.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:29.771+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:30.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:29.955+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:30.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:29.955+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:30.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:29.955+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:30.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:29.956+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:30.671 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:30.286+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:30.286+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:30.286+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:30.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:30.286+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:31.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:30.814+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:31.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:30.814+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:31.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:30.814+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:31.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:30.815+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:31.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:30.924+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:31.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:30.924+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:31.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:30.924+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:31.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:30.924+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:31.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:31.317+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:31.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:31.317+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:31.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:31.317+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:31.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:31.318+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:31.902 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:31.793+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:31.902 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:31.793+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:31.902 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:31.793+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:31.903 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:31.794+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:32.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:31.903+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:32.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:31.903+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:32.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:31.903+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:32.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:31.903+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:32.297+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:32.297+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:32.297+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:32.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:32.298+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:33.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:32.770+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:33.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:32.770+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:33.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:32.770+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:33.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:32.771+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:33.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:32.924+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:33.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:32.924+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:33.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:32.924+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:33.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:32.925+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:33.606 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:33.303+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:33.606 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:33.303+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:33.606 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:33.303+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:33.606 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:33.304+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:33.878 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:26:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:26:33.690+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13084 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:26:33.878 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:33.769+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:33.878 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:33.769+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:33.878 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:33.769+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:33.879 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:33.769+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:33.879 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:33.878+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:33.879 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:33.878+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:33.879 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:33.878+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:34.171 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:33.879+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:34.575 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:34.293+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:34.575 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:34.293+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:34.576 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:34.293+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:34.576 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:34.294+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:34.846 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:34.818+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:34.847 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:34.818+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:34.847 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:34.818+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:34.847 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:34.818+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:34.847+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:34.847+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:34.847+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:35.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:34.847+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:35.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:35.248+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:35.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:35.248+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:35.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:35.248+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:35.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:35.249+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:36.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:35.883+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:36.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:35.883+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:36.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:35.883+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:36.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:35.884+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:36.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:35.863+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:36.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:35.863+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:36.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:35.863+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:36.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:35.863+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:36.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:36.291+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:36.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:36.291+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:36.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:36.291+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:36.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:36.293+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:37.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:36.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:37.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:36.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:37.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:36.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:37.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:36.872+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:37.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:36.899+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:37.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:36.899+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:37.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:36.899+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:37.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:36.900+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:37.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:37.283+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:37.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:37.283+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:37.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:37.283+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:37.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:37.284+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:38.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:37.828+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:38.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:37.828+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:38.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:37.828+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:38.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:37.828+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:38.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:37.914+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:38.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:37.914+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:38.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:37.914+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:38.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:37.915+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:38.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:38.323+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:38.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:38.323+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:38.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:38.323+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:38.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:38.325+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:39.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:26:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:26:38.691+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13137 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:26:39.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:38.839+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:39.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:38.839+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:39.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:38.839+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:39.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:38.840+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:39.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:38.907+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:39.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:38.907+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:39.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:38.907+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:39.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:38.907+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:39.611 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:39.317+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:39.612 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:39.317+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:39.612 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:39.317+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:39.612 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:39.318+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:39.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:39.816+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:39.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:39.816+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:39.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:39.816+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:39.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:39.817+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:40.296 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:39.940+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:40.296 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:39.940+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:40.296 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:39.940+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:40.296 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:39.940+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:40.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:40.296+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:40.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:40.296+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:40.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:40.296+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:40.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:40.297+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:41.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:40.969+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:41.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:40.969+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:41.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:40.969+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:41.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:40.970+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:41.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:40.771+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:41.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:40.771+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:41.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:40.771+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:41.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:40.772+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:41.569 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:41.274+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:41.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:41.274+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:41.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:41.274+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:41.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:41.274+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:41.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:41.788+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:41.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:41.788+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:41.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:41.788+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:41.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:41.789+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:42.277 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:42.007+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:42.277 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:42.007+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:42.277 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:42.007+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:42.277 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:42.008+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:42.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:42.278+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:42.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:42.278+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:42.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:42.278+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:42.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:42.279+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:43.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:42.980+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:43.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:42.980+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:43.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:42.980+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:43.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:42.981+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:43.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:42.826+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:43.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:42.826+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:43.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:42.826+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:43.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:42.827+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:43.606 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:43.257+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:43.607 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:43.257+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:43.607 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:43.257+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:43.607 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:43.257+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:43.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:26:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:26:43.691+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13188 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:26:43.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:43.784+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:43.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:43.784+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:43.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:43.784+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:43.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:43.785+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:44.302 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:44.011+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:44.303 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:44.011+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:44.303 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:44.011+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:44.303 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:44.012+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:44.647 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:44.302+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:44.648 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:44.302+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:44.648 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:44.302+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:44.648 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:44.303+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:44.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:44.752+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:44.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:44.752+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:44.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:44.752+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:44.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:44.752+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:45.338 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:45.010+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:45.338 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:45.010+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:45.339 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:45.010+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:45.339 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:45.011+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:45.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:45.338+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:45.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:45.338+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:45.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:45.338+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:45.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:45.338+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:46.058 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:45.751+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:46.058 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:45.751+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:46.058 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:45.751+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:46.058 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:45.752+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:46.348 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:46.057+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:46.348 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:46.057+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:46.348 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:46.057+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:46.348 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:46.058+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:46.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:46.347+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:46.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:46.347+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:46.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:46.347+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:46.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:46.348+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:47.084 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:46.791+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:47.084 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:46.791+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:47.084 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:46.791+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:47.085 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:46.792+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:47.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:47.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:47.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:47.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:47.084+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:47.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:47.310+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:47.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:47.310+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:47.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:47.310+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:47.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:47.311+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:48.116 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:47.783+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:48.116 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:47.783+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:48.116 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:47.783+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:48.116 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:47.784+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:48.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:48.263+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:48.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:48.263+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:48.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:48.263+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:48.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:48.265+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:48.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:48.115+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:48.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:48.115+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:48.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:48.115+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:48.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:48.116+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:49.079 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:26:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:26:48.692+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13241 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:26:49.079 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:48.803+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:49.080 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:48.803+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:49.080 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:48.803+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:49.080 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:48.803+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:49.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:49.078+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:49.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:49.078+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:49.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:49.078+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:49.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:49.079+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:49.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:49.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:49.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:49.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:49.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:49.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:49.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:49.236+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:49.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:49.767+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:49.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:49.767+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:49.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:49.767+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:49.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:49.767+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:50.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:50.110+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:50.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:50.110+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:50.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:50.110+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:50.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:50.112+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:50.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:50.188+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:50.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:50.188+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:50.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:50.188+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:50.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:50.189+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:51.083 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:50.734+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:51.084 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:50.734+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:51.084 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:50.734+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:51.084 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:50.735+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:51.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:51.197+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:51.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:51.197+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:51.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:51.197+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:51.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:51.198+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:51.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:51.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:51.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:51.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:51.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:51.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:51.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:51.084+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:51.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:51.765+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:51.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:51.765+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:51.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:51.765+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:51.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:51.766+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:52.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:52.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:52.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:52.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:52.119+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:52.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:52.235+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:52.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:52.235+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:52.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:52.235+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:52.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:52.235+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:53.120 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:52.777+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:53.120 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:52.777+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:53.120 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:52.777+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:53.120 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:52.778+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:53.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:53.119+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:53.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:53.119+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:53.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:53.119+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:53.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:53.120+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:53.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:53.196+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:53.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:53.196+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:53.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:53.196+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:53.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:53.196+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:53.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:26:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:26:53.694+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13293 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:26:53.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:53.756+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:53.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:53.756+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:53.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:53.756+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:53.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:53.756+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:54.122+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:54.122+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:54.122+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:54.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:54.123+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:54.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:54.231+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:54.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:54.231+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:54.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:54.231+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:54.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:54.232+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:55.160 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:54.720+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:55.160 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:54.720+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:55.160 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:54.720+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:55.160 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:54.721+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:55.206+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:55.206+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:55.206+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:55.207+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:55.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:55.159+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:55.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:55.159+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:55.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:55.159+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:55.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:55.160+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:56.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:55.695+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:56.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:55.695+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:56.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:55.695+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:56.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:55.695+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:56.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:56.197+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:56.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:56.197+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:56.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:56.197+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:56.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:56.198+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:56.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:56.181+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:56.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:56.181+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:56.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:56.181+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:56.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:56.182+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:57.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:56.685+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:57.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:56.685+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:57.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:56.685+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:57.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:56.686+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:57.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:57.193+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:57.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:57.193+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:57.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:57.193+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:57.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:57.194+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:57.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:57.222+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:57.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:57.222+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:57.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:57.222+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:57.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:57.223+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:58.162 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:57.719+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:58.162 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:57.719+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:58.163 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:57.719+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:58.163 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:57.720+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:58.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:58.257+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:58.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:58.257+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:58.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:58.257+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:58.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:58.258+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:58.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:58.163+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:58.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:58.163+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:58.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:58.163+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:58.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:58.163+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:26:59.124 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:26:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:26:58.696+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13343 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:26:59.124 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:58.733+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:59.124 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:58.733+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:59.125 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:58.733+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:26:59.125 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:58.735+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:59.267+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:59.267+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:59.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:59.267+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:26:59.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:26:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:26:59.268+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:26:59.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:59.124+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:59.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:59.124+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:59.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:59.124+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:26:59.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:26:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:26:59.125+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:00.089 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:59.753+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:00.090 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:59.753+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:00.090 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:59.753+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:00.090 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:26:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:26:59.753+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:00.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:00.315+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:00.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:00.315+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:00.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:00.315+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:00.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:00.316+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:00.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:00.090+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:00.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:00.090+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:00.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:00.090+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:00.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:00.091+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:01.128 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:00.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:01.128 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:00.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:01.129 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:00.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:01.129 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:00.770+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:01.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:01.128+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:01.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:01.128+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:01.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:01.128+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:01.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:01.129+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:01.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:01.360+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:01.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:01.360+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:01.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:01.360+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:01.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:01.361+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:01.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:01.800+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:01.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:01.800+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:01.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:01.800+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:01.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:01.801+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:02.359 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:02.092+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:02.359 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:02.092+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:02.359 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:02.092+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:02.360 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:02.093+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:02.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:02.359+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:02.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:02.359+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:02.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:02.359+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:02.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:02.360+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:03.088 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:02.795+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:03.089 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:02.795+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:03.089 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:02.795+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:03.089 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:02.795+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:03.381 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:03.088+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:03.381 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:03.088+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:03.382 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:03.088+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:03.382 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:03.089+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:03.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:03.381+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:03.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:03.381+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:03.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:03.381+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:03.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:03.383+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:04.117 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:27:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:27:03.697+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13386 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:27:04.117 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:03.750+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:04.118 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:03.750+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:04.118 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:03.750+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:04.118 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:03.751+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:04.367 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:04.117+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:04.368 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:04.117+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:04.368 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:04.117+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:04.368 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:04.119+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:04.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:04.368+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:04.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:04.368+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:04.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:04.368+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:04.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:04.368+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:05.168 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:04.776+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:05.168 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:04.776+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:05.168 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:04.776+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:05.168 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:04.777+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:05.408+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:05.408+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:05.408+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:05.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:05.409+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:05.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:05.168+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:05.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:05.168+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:05.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:05.168+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:05.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:05.169+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:05.826+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:05.826+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:05.826+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:05.826+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:06.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:06.216+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:06.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:06.216+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:06.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:06.216+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:06.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:06.217+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:06.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:06.425+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:06.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:06.425+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:06.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:06.425+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:06.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:06.426+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:07.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:06.869+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:07.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:06.869+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:07.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:06.869+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:07.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:06.870+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:07.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:07.388+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:07.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:07.388+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:07.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:07.388+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:07.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:07.389+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:07.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:07.177+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:07.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:07.177+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:07.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:07.177+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:07.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:07.178+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:08.147 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:07.859+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:08.147 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:07.859+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:08.147 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:07.859+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:08.147 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:07.860+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:08.147+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:08.147+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:08.147+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:08.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:08.148+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:08.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:08.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:08.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:08.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:08.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:08.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:08.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:08.375+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:09.109 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:27:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:27:08.699+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13429 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:27:09.109 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:08.896+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:09.109 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:08.896+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:09.109 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:08.896+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:09.109 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:08.897+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:09.413 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:09.109+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:09.414 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:09.109+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:09.414 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:09.109+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:09.414 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:09.110+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:09.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:09.413+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:09.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:09.414+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:09.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:09.414+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:09.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:09.415+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:10.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:09.926+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:10.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:09.926+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:10.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:09.926+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:10.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:09.926+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:10.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:10.066+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:10.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:10.066+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:10.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:10.066+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:10.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:10.066+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:10.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:10.394+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:10.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:10.394+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:10.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:10.394+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:10.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:10.395+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:11.417 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:10.960+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:11.417 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:10.960+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:11.417 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:10.960+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:11.418 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:10.961+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:11.418 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:11.073+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:11.418 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:11.073+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:11.418 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:11.073+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:11.418 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:11.073+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:11.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:11.417+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:11.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:11.417+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:11.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:11.417+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:11.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:11.418+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:12.401 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:11.982+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:12.402 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:11.982+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:12.402 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:11.982+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:12.402 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:11.982+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:12.402 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:12.094+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:12.402 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:12.094+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:12.402 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:12.094+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:12.403 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:12.095+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:12.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:12.401+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:12.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:12.401+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:12.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:12.401+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:12.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:12.403+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:13.406 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:13.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:13.406 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:13.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:13.406 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:13.023+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:13.407 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:13.024+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:13.407 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:13.141+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:13.407 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:13.141+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:13.407 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:13.141+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:13.407 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:13.142+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:13.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:13.406+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:13.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:13.406+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:13.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:13.406+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:13.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:13.407+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:14.035 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:27:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:27:13.700+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13471 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:27:14.396 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:14.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:14.396 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:14.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:14.396 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:14.035+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:14.396 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:14.036+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:14.397 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:14.137+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:14.397 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:14.137+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:14.397 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:14.137+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:14.397 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:14.138+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:14.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:14.396+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:14.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:14.396+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:14.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:14.396+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:14.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:14.397+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:15.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:15.105+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:15.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:15.105+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:15.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:15.105+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:15.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:15.105+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:15.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:15.075+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:15.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:15.075+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:15.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:15.075+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:15.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:15.077+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:15.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:15.376+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:15.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:15.376+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:15.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:15.376+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:15.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:15.377+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:16.366 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:16.124+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:16.366 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:16.124+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:16.366 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:16.124+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:16.366 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:16.125+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:16.367 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:16.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:16.367 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:16.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:16.367 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:16.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:16.367 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:16.073+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:16.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:16.366+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:16.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:16.366+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:16.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:16.366+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:16.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:16.367+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:17.396 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:17.165+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:17.396 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:17.165+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:17.396 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:17.165+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:17.396 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:17.167+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:17.396 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:17.095+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:17.397 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:17.095+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:17.397 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:17.095+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:17.397 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:17.095+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:17.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:17.396+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:17.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:17.396+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:17.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:17.396+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:17.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:17.397+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:18.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:18.150+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:18.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:18.150+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:18.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:18.150+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:18.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:18.151+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:18.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:18.063+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:18.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:18.063+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:18.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:18.063+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:18.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:18.065+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:18.700 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:18.440+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:18.701 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:18.440+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:18.701 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:18.440+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:18.701 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:18.441+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:19.015 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:27:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:27:18.701+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13515 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:27:19.405 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:19.015+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:19.405 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:19.015+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:19.405 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:19.015+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:19.406 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:19.016+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:19.406 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:19.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:19.406 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:19.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:19.406 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:19.183+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:19.406 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:19.184+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:19.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:19.405+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:19.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:19.405+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:19.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:19.405+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:19.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:19.406+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:20.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:20.003+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:20.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:20.003+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:20.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:20.003+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:20.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:20.004+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:20.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:20.384+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:20.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:20.384+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:20.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:20.384+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:20.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:20.384+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:20.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:20.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:20.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:20.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:20.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:20.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:20.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:20.173+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:21.384 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:21.213+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:21.384 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:21.213+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:21.384 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:21.213+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:21.384 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:21.213+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:21.385 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:21.006+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:21.385 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:21.006+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:21.385 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:21.006+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:21.385 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:21.006+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:21.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:21.384+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:21.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:21.384+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:21.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:21.384+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:21.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:21.385+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:22.382 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:22.238+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:22.383 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:22.238+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:22.383 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:22.238+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:22.383 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:22.239+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:22.383 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:22.016+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:22.383 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:22.016+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:22.383 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:22.016+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:22.384 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:22.018+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:22.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:22.383+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:22.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:22.383+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:22.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:22.383+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:22.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:22.384+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:23.255 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:23.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:23.256 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:23.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:23.256 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:23.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:23.256 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:23.000+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:23.610 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:23.256+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:23.610 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:23.256+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:23.610 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:23.256+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:23.611 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:23.257+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:23.611 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:23.397+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:23.611 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:23.397+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:23.611 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:23.397+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:23.611 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:23.398+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:23.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:27:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:27:23.703+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13561 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:27:24.287 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:24.030+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:24.287 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:24.030+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:24.287 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:24.030+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:24.287 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:24.031+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:24.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:24.386+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:24.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:24.386+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:24.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:24.386+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:24.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:24.386+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:24.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:24.287+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:24.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:24.287+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:24.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:24.287+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:24.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:24.288+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:25.290 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:25.031+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:25.290 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:25.031+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:25.290 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:25.031+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:25.290 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:25.032+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:25.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:25.403+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:25.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:25.403+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:25.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:25.403+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:25.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:25.403+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:25.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:25.290+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:25.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:25.290+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:25.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:25.290+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:25.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:25.291+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:26.268 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:26.006+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:26.268 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:26.006+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:26.268 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:26.006+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:26.268 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:26.007+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:26.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:26.447+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:26.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:26.447+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:26.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:26.447+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:26.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:26.448+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:26.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:26.268+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:26.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:26.268+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:26.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:26.268+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:26.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:26.269+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:27.298 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:27.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:27.298 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:27.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:27.298 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:27.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:27.299 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:27.034+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:27.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:27.467+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:27.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:27.467+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:27.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:27.467+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:27.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:27.468+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:27.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:27.298+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:27.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:27.298+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:27.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:27.298+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:27.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:27.299+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:28.337 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:28.030+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:28.337 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:28.030+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:28.337 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:28.030+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:28.337 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:28.030+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:28.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:28.337+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:28.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:28.337+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:28.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:28.337+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:28.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:28.338+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:28.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:28.472+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:28.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:28.472+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:28.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:28.472+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:28.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:28.473+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:29.005 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:27:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:27:28.704+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13613 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:27:29.306 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:29.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:29.306 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:29.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:29.306 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:29.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:29.306 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:29.006+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:29.557 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:29.306+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:29.557 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:29.306+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:29.558 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:29.306+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:29.558 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:29.307+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:29.558 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:29.480+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:29.558 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:29.480+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:29.558 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:29.480+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:29.558 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:29.481+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:30.265 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:29.974+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:30.266 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:29.974+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:30.266 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:29.974+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:30.266 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:29.975+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:30.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:30.441+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:30.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:30.441+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:30.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:30.441+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:30.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:30.442+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:30.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:30.265+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:30.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:30.265+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:30.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:30.265+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:30.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:30.267+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:31.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:31.004+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:31.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:31.004+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:31.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:31.004+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:31.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:31.005+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:31.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:31.219+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:31.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:31.219+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:31.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:31.219+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:31.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:31.220+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:31.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:31.424+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:31.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:31.424+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:31.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:31.424+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:31.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:31.425+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:32.412 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:32.025+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:32.412 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:32.025+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:32.412 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:32.025+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:32.413 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:32.026+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:32.413 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:32.197+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:32.413 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:32.197+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:32.413 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:32.197+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:32.413 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:32.197+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:32.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:32.412+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:32.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:32.412+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:32.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:32.412+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:32.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:32.414+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:33.368 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:33.195+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:33.368 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:33.195+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:33.368 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:33.195+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:33.368 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:33.196+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:33.369 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:33.069+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:33.369 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:33.069+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:33.369 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:33.069+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:33.369 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:33.069+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:33.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:33.368+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:33.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:33.368+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:33.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:33.368+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:33.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:33.369+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:34.072 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:27:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:27:33.705+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13664 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:27:34.362 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:34.218+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:34.362 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:34.218+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:34.362 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:34.218+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:34.363 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:34.219+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:34.363 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:34.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:34.363 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:34.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:34.363 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:34.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:34.363 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:34.073+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:34.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:34.362+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:34.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:34.362+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:34.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:34.362+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:34.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:34.363+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:35.366 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:35.026+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:35.367 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:35.026+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:35.367 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:35.026+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:35.367 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:35.027+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:35.367 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:35.187+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:35.367 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:35.187+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:35.367 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:35.187+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:35.368 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:35.189+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:35.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:35.366+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:35.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:35.366+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:35.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:35.367+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:35.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:35.368+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:36.378 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:36.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:36.378 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:36.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:36.378 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:36.072+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:36.378 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:36.073+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:36.378 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:36.147+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:36.379 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:36.147+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:36.379 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:36.147+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:36.379 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:36.148+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:36.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:36.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:36.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:36.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:36.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:36.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:36.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:36.379+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:37.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:37.032+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:37.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:37.032+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:37.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:37.032+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:37.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:37.032+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:37.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:37.103+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:37.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:37.103+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:37.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:37.103+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:37.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:37.103+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:37.714 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:37.428+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:37.714 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:37.428+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:37.714 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:37.428+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:37.715 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:37.428+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:38.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:38.076+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:38.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:38.076+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:38.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:38.076+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:38.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:38.077+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:38.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:38.098+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:38.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:38.098+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:38.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:38.098+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:38.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:38.099+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:38.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:27:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:27:38.707+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13717 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:27:38.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:38.462+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:38.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:38.462+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:38.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:38.462+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:38.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:38.463+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:39.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:39.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:39.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:39.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:39.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:39.068+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:39.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:39.068+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:39.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:39.105+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:39.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:39.105+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:39.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:39.105+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:39.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:39.105+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:39.797 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:39.429+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:39.798 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:39.429+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:39.798 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:39.429+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:39.798 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:39.430+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:40.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:40.113+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:40.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:40.113+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:40.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:40.113+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:40.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:40.115+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:40.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:40.056+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:40.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:40.056+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:40.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:40.056+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:40.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:40.057+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:40.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:40.436+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:40.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:40.436+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:40.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:40.436+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:40.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:40.438+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:41.130 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:41.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:41.131 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:41.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:41.131 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:41.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:41.131 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:41.066+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:41.403 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:41.129+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:41.404 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:41.129+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:41.404 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:41.129+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:41.404 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:41.130+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:41.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:41.404+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:41.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:41.404+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:41.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:41.404+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:41.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:41.405+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:42.159 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:42.065+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:42.159 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:42.065+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:42.159 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:42.065+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:42.159 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:42.065+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:42.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:42.363+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:42.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:42.363+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:42.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:42.363+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:42.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:42.365+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:42.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:42.158+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:42.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:42.158+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:42.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:42.158+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:42.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:42.159+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:43.397 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:43.153+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:43.397 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:43.153+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:43.397 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:43.153+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:43.398 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:43.154+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:43.398 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:43.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:43.398 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:43.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:43.398 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:43.029+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:43.398 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:43.030+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:43.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:43.397+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:43.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:43.397+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:43.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:43.397+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:43.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:43.398+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:44.058 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:27:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:27:43.708+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13769 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:27:44.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:44.181+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:44.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:44.181+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:44.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:44.181+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:44.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:44.182+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:44.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:44.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:44.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:44.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:44.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:44.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:44.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:44.059+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:44.917 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:44.435+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:44.917 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:44.435+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:44.917 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:44.435+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:44.917 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:44.435+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:45.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:45.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:45.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:45.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:45.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:45.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:45.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:45.050+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:45.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:45.414+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:45.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:45.414+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:45.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:45.414+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:45.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:45.415+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:45.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:45.210+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:45.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:45.210+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:45.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:45.210+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:45.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:45.210+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:46.390 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:46.168+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:46.390 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:46.168+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:46.390 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:46.168+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:46.390 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:46.168+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:46.390 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:46.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:46.390 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:46.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:46.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:46.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:46.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:46.022+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:46.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:46.389+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:46.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:46.389+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:46.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:46.389+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:46.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:46.391+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:47.380 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:47.202+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:47.380 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:47.202+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:47.381 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:47.202+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:47.381 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:47.203+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:47.381 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:47.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:47.381 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:47.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:47.381 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:47.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:47.381 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:47.072+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:47.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:47.380+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:47.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:47.380+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:47.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:47.380+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:47.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:47.381+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:48.403 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:48.227+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:48.403 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:48.227+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:48.403 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:48.227+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:48.403 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:48.228+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:48.403 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:48.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:48.404 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:48.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:48.404 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:48.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:48.404 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:48.023+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:48.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:48.403+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:48.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:48.403+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:48.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:48.403+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:48.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:48.403+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:49.027 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:27:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:27:48.710+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13822 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:27:49.370 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:49.028+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:49.370 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:49.028+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:49.370 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:49.028+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:49.370 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:49.028+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:49.370 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:49.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:49.371 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:49.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:49.371 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:49.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:49.371 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:49.205+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:49.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:49.369+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:49.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:49.369+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:49.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:49.369+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:49.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:49.371+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:50.363 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:50.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:50.363 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:50.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:50.363 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:50.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:50.363 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:50.183+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:50.364 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:50.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:50.364 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:50.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:50.364 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:50.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:50.364 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:50.017+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:50.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:50.363+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:50.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:50.363+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:50.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:50.363+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:50.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:50.364+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:51.375 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:51.009+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:51.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:51.009+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:51.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:51.009+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:51.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:51.010+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:51.376 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:51.212+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:51.376 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:51.212+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:51.377 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:51.212+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:51.377 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:51.213+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:51.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:51.375+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:51.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:51.375+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:51.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:51.375+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:51.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:51.376+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:52.363 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:52.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:52.363 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:52.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:52.363 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:52.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:52.363 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:52.025+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:52.364 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:52.222+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:52.364 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:52.222+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:52.364 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:52.222+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:52.364 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:52.222+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:52.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:52.363+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:52.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:52.363+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:52.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:52.363+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:52.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:52.364+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:53.364 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:53.031+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:53.365 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:53.031+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:53.365 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:53.031+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:53.365 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:53.031+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:53.365 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:53.265+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:53.365 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:53.265+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:53.366 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:53.265+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:53.366 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:53.266+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:53.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:53.365+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:53.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:53.365+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:53.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:53.365+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:53.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:53.366+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:54.024 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:27:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:27:53.711+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13874 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:27:54.298 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:54.024+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:54.299 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:54.024+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:54.299 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:54.024+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:54.299 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:54.025+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:54.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:54.298+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:54.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:54.298+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:54.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:54.298+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:54.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:54.300+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:54.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:54.378+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:54.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:54.378+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:54.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:54.378+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:54.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:54.379+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:55.283 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:54.988+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:55.283 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:54.988+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:55.283 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:54.988+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:55.283 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:54.989+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:55.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:55.425+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:55.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:55.425+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:55.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:55.425+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:55.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:55.425+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:55.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:55.283+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:55.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:55.283+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:55.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:55.283+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:55.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:55.284+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:56.409 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:56.245+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:56.409 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:56.245+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:56.409 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:56.245+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:56.409 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:56.246+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:56.409 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:56.037+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:56.409 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:56.037+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:56.410 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:56.037+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:56.410 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:56.038+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:56.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:56.408+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:56.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:56.408+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:56.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:56.408+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:56.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:56.410+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:57.406 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:57.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:57.406 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:57.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:57.406 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:57.204+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:57.406 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:57.206+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:57.406 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:57.040+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:57.407 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:57.040+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:57.407 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:57.040+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:57.407 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:57.040+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:57.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:57.406+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:57.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:57.406+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:57.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:57.406+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:57.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:57.406+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:58.393 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:58.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:58.393 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:58.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:58.393 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:58.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:58.393 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:58.237+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:58.394 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:58.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:58.394 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:58.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:58.394 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:58.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:58.394 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:58.079+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:58.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:58.393+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:58.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:58.393+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:58.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:58.393+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:58.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:58.393+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:59.101 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:27:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:27:58.712+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13927 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:27:59.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:59.217+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:59.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:59.217+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:59.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:59.217+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:27:59.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:27:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:27:59.218+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:27:59.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:59.101+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:59.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:59.101+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:59.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:59.101+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:27:59.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:27:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:27:59.102+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:27:59.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:59.348+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:59.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:59.348+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:59.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:59.348+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:27:59.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:27:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:27:59.349+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:00.350 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:00.220+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:00.350 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:00.220+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:00.350 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:00.220+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:00.351 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:00.221+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:00.351 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:00.093+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:00.351 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:00.093+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:00.351 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:00.093+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:00.351 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:00.094+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:00.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:00.350+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:00.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:00.350+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:00.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:00.350+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:00.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:00.351+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:01.335 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:01.332+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:01.335 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:01.332+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:01.335 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:01.332+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:01.335 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:01.333+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:01.336 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:01.082+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:01.336 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:01.082+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:01.336 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:01.082+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:01.336 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:01.083+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:01.336 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:01.253+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:01.336 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:01.253+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:01.336 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:01.253+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:01.337 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:01.253+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:02.340 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:02.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:02.340 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:02.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:02.340 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:02.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:02.340 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:02.037+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:02.340 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:02.219+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:02.341 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:02.219+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:02.341 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:02.219+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:02.341 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:02.220+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:02.347 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-27T04:28:02.347 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-04-27T04:28:02.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:28:02.348 DEBUG:teuthology.orchestra.run.smithi046:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-04-27T04:28:02.382 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc: Cannot mkdir: Input/output error 2024-04-27T04:28:02.382 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3: Cannot mkdir: Input/output error 2024-04-27T04:28:02.382 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/random_range.3: Cannot open: Input/output error 2024-04-27T04:28:02.383 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/str_to_bytes.3: Cannot open: Input/output error 2024-04-27T04:28:02.383 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/usctest.3: Cannot open: Input/output error 2024-04-27T04:28:02.383 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/string_to_tokens.3: Cannot open: Input/output error 2024-04-27T04:28:02.383 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/Makefile: Cannot open: Input/output error 2024-04-27T04:28:02.383 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/pattern.3: Cannot open: Input/output error 2024-04-27T04:28:02.383 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/tst_tmpdir.3: Cannot open: Input/output error 2024-04-27T04:28:02.383 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/forker.3: Cannot open: Input/output error 2024-04-27T04:28:02.383 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/rmobj.3: Cannot open: Input/output error 2024-04-27T04:28:02.383 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/random_range_seed.3: Cannot open: Input/output error 2024-04-27T04:28:02.384 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/parse_open_flags.3: Cannot open: Input/output error 2024-04-27T04:28:02.384 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/parse_opts.3: Cannot open: Input/output error 2024-04-27T04:28:02.384 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/write_log.3: Cannot open: Input/output error 2024-04-27T04:28:02.384 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/parse_ranges.3: Cannot open: Input/output error 2024-04-27T04:28:02.384 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/tst_set_error.3: Cannot open: Input/output error 2024-04-27T04:28:02.384 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/tst_res.3: Cannot open: Input/output error 2024-04-27T04:28:02.384 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/tst_sig.3: Cannot open: Input/output error 2024-04-27T04:28:02.384 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man3/get_attrib.3: Cannot open: Input/output error 2024-04-27T04:28:02.385 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man1: Cannot mkdir: Input/output error 2024-04-27T04:28:02.385 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man1/ltp-bump.1: Cannot open: Input/output error 2024-04-27T04:28:02.385 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man1/doio.1: Cannot open: Input/output error 2024-04-27T04:28:02.385 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man1/Makefile: Cannot open: Input/output error 2024-04-27T04:28:02.385 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man1/ltp-pan.1: Cannot open: Input/output error 2024-04-27T04:28:02.385 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/man1/iogen.1: Cannot open: Input/output error 2024-04-27T04:28:02.385 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/testcases: Cannot mkdir: Input/output error 2024-04-27T04:28:02.391 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/testcases/kernel.txt: Cannot open: Input/output error 2024-04-27T04:28:02.391 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/testcases/misc.txt: Cannot open: Input/output error 2024-04-27T04:28:02.391 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/testcases/network.txt: Cannot open: Input/output error 2024-04-27T04:28:02.391 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/testcases/commands.txt: Cannot open: Input/output error 2024-04-27T04:28:02.391 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/testcases/open_posix_testsuite.txt: Cannot open: Input/output error 2024-04-27T04:28:02.392 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/testcases/open_hpi_testsuite.txt: Cannot open: Input/output error 2024-04-27T04:28:02.392 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/ltp-howto.lyx: Cannot open: Input/output error 2024-04-27T04:28:02.392 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/Makefile: Cannot open: Input/output error 2024-04-27T04:28:02.392 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/automation-cookbook.html: Cannot open: Input/output error 2024-04-27T04:28:02.392 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/examples: Cannot mkdir: Input/output error 2024-04-27T04:28:02.392 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/examples/quickhit.c: Cannot open: Input/output error 2024-04-27T04:28:02.392 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/examples/sudoers: Cannot open: Input/output error 2024-04-27T04:28:02.392 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/examples/super.tab: Cannot open: Input/output error 2024-04-27T04:28:02.392 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/ltp-run-files.txt: Cannot open: Input/output error 2024-04-27T04:28:02.393 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/tpm-commands.txt: Cannot open: Input/output error 2024-04-27T04:28:02.393 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/ltp-howto.txt: Cannot open: Input/output error 2024-04-27T04:28:02.393 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/network_stress.txt: Cannot open: Input/output error 2024-04-27T04:28:02.393 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/doc/automation-README.html: Cannot open: Input/output error 2024-04-27T04:28:02.393 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4: Cannot mkdir: Input/output error 2024-04-27T04:28:02.393 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/ltp-cgroupstats.m4: Cannot open: Input/output error 2024-04-27T04:28:02.393 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/ltp-signalfd.m4: Cannot open: Input/output error 2024-04-27T04:28:02.393 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/GNUmakefile: Cannot open: Input/output error 2024-04-27T04:28:02.394 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/ltp-eventfd.m4: Cannot open: Input/output error 2024-04-27T04:28:02.394 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/ltp-quota.m4: Cannot open: Input/output error 2024-04-27T04:28:02.394 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/ltp-version.m4: Cannot open: Input/output error 2024-04-27T04:28:02.394 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/ltp-cap.m4: Cannot open: Input/output error 2024-04-27T04:28:02.394 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/ltp-selinux.m4: Cannot open: Input/output error 2024-04-27T04:28:02.394 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/ltp-signal.m4: Cannot open: Input/output error 2024-04-27T04:28:02.394 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/ltp-ptrace.m4: Cannot open: Input/output error 2024-04-27T04:28:02.394 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/ltp-numa.m4: Cannot open: Input/output error 2024-04-27T04:28:02.395 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/Makefile.in: Cannot open: Input/output error 2024-04-27T04:28:02.395 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/ltp-modify_ldt.m4: Cannot open: Input/output error 2024-04-27T04:28:02.395 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/ltp-unshare.m4: Cannot open: Input/output error 2024-04-27T04:28:02.395 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/ltp-taskstats.m4: Cannot open: Input/output error 2024-04-27T04:28:02.395 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/ltp-crypto.m4: Cannot open: Input/output error 2024-04-27T04:28:02.395 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/m4/Makefile.am: Cannot open: Input/output error 2024-04-27T04:28:02.395 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/runltp: Cannot open: Input/output error 2024-04-27T04:28:02.395 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/Makefile: Cannot open: Input/output error 2024-04-27T04:28:02.395 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/configure: Cannot open: Input/output error 2024-04-27T04:28:02.396 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/config.guess: Cannot open: Input/output error 2024-04-27T04:28:02.396 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/execltp.in: Cannot open: Input/output error 2024-04-27T04:28:02.396 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib: Cannot mkdir: Input/output error 2024-04-27T04:28:02.396 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/rmobj.c: Cannot open: Input/output error 2024-04-27T04:28:02.397 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/datapid.c: Cannot open: Input/output error 2024-04-27T04:28:02.397 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/tst_kvercmp.c: Cannot open: Input/output error 2024-04-27T04:28:02.397 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/tst_sig.c: Cannot open: Input/output error 2024-04-27T04:28:02.397 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/tst_cwd_has_free.c: Cannot open: Input/output error 2024-04-27T04:28:02.397 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/parse_opts.c: Cannot open: Input/output error 2024-04-27T04:28:02.397 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/tlibio.c: Cannot open: Input/output error 2024-04-27T04:28:02.397 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/cloner.c: Cannot open: Input/output error 2024-04-27T04:28:02.397 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/system_specific_process_info.c: Cannot open: Input/output error 2024-04-27T04:28:02.398 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/tst_tmpdir.c: Cannot open: Input/output error 2024-04-27T04:28:02.398 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/open_flags.c: Cannot open: Input/output error 2024-04-27T04:28:02.398 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/get_high_address.c: Cannot open: Input/output error 2024-04-27T04:28:02.398 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/Makefile: Cannot open: Input/output error 2024-04-27T04:28:02.398 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/write_log.c: Cannot open: Input/output error 2024-04-27T04:28:02.398 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/file_lock.c: Cannot open: Input/output error 2024-04-27T04:28:02.398 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/tst_is_cwd.c: Cannot open: Input/output error 2024-04-27T04:28:02.398 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/string_to_tokens.c: Cannot open: Input/output error 2024-04-27T04:28:02.399 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/system_specific_hugepages_info.c: Cannot open: Input/output error 2024-04-27T04:28:02.399 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/str_to_bytes.c: Cannot open: Input/output error 2024-04-27T04:28:02.399 INFO:tasks.workunit.client.0.smithi046.stderr:tar: 2024-04-27T04:28:02.399 INFO:tasks.workunit.client.0.smithi046.stderr:gzip: ltp-full-20091231/lib/random_range.c: Cannot open: Input/output error 2024-04-27T04:28:02.399 INFO:tasks.workunit.client.0.smithi046.stderr:stdin: Stale file handle 2024-04-27T04:28:02.399 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/libtestsuite.c: Cannot open: Input/output error 2024-04-27T04:28:02.399 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/ltp.pc.in: Cannot open: Input/output error 2024-04-27T04:28:02.400 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231/lib/tst_res.c: Cannot open: Input/output error 2024-04-27T04:28:02.400 INFO:tasks.workunit.client.0.smithi046.stderr:tar: Unexpected EOF in archive 2024-04-27T04:28:02.400 INFO:tasks.workunit.client.0.smithi046.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-04-27T04:28:02.400 INFO:tasks.workunit.client.0.smithi046.stderr:tar: Error is not recoverable: exiting now 2024-04-27T04:28:02.401 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-27T04:28:02.401 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-04-27T04:28:02.401 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-27T04:28:02.402 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-04-27T04:28:02.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T04:28:02.402 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-27T04:28:02.478 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-04-27T04:28:02.478 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T04:28:02.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:02.339+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:02.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:02.339+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:02.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:02.339+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:02.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:02.340+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:03.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:02.990+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:03.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:02.990+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:03.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:02.990+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:03.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:02.990+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:03.612 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:03.388+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:03.612 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:03.388+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:03.612 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:03.388+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:03.612 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:03.389+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:03.613 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:03.266+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:03.613 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:03.266+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:03.613 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:03.266+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:03.613 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:03.267+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:03.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:28:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:28:03.714+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 13978 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:28:04.284 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:03.948+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:04.285 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:03.948+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:04.285 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:03.948+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:04.285 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:03.949+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:04.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:04.436+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:04.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:04.436+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:04.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:04.436+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:04.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:04.437+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:04.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:04.284+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:04.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:04.284+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:04.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:04.284+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:04.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:04.285+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:05.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:04.921+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:05.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:04.921+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:05.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:04.921+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:05.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:04.922+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:05.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:05.309+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:05.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:05.309+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:05.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:05.309+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:05.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:05.310+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:05.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:05.420+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:05.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:05.420+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:05.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:05.420+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:05.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:05.421+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:05.903+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:06.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:05.903+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:06.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:05.903+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:06.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:05.904+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:06.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:06.393+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:06.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:06.393+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:06.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:06.393+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:06.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:06.393+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:06.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:06.299+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:06.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:06.299+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:06.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:06.299+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:06.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:06.300+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:07.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:06.904+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:07.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:06.904+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:07.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:06.904+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:07.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:06.905+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:07.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:07.386+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:07.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:07.386+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:07.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:07.386+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:07.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:07.387+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:07.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:07.253+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:07.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:07.253+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:07.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:07.253+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:07.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:07.254+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:08.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:07.900+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:08.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:07.900+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:08.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:07.900+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:08.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:07.900+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:08.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:08.405+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:08.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:08.405+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:08.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:08.405+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:08.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:08.406+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:08.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:08.233+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:08.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:08.233+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:08.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:08.233+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:08.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:08.234+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:09.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:28:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:28:08.715+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14029 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:28:09.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:08.856+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:09.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:08.856+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:09.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:08.856+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:09.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:08.858+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:09.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:09.379+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:09.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:09.379+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:09.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:09.379+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:09.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:09.380+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:09.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:09.223+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:09.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:09.223+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:09.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:09.223+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:09.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:09.224+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:10.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:09.828+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:10.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:09.828+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:10.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:09.828+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:10.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:09.828+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:10.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:10.333+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:10.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:10.333+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:10.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:10.333+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:10.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:10.334+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:10.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:10.206+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:10.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:10.207+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:10.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:10.207+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:10.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:10.208+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:11.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:10.822+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:11.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:10.822+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:11.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:10.822+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:11.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:10.823+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:11.570 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:11.336+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:11.570 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:11.336+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:11.570 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:11.336+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:11.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:11.338+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:11.571 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:11.175+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:11.571 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:11.175+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:11.571 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:11.175+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:11.571 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:11.176+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:11.869+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:12.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:11.869+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:12.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:11.869+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:12.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:11.869+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:12.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:12.385+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:12.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:12.385+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:12.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:12.385+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:12.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:12.386+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:12.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:12.197+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:12.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:12.197+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:12.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:12.197+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:12.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:12.198+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:13.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:12.882+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:13.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:12.882+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:13.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:12.882+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:13.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:12.883+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:13.613 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:13.362+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:13.613 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:13.362+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:13.613 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:13.362+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:13.613 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:13.362+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:13.614 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:13.224+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:13.614 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:13.224+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:13.614 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:13.224+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:13.614 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:13.225+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:13.864 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:28:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:28:13.716+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14081 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:28:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:13.864+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:13.864+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:13.864+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:14.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:13.865+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:14.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:14.404+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:14.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:14.404+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:14.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:14.404+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:14.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:14.404+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:14.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:14.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:14.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:14.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:14.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:14.182+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:14.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:14.183+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:15.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:14.842+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:15.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:14.842+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:15.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:14.842+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:15.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:14.843+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:15.637 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:15.190+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:15.637 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:15.190+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:15.638 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:15.190+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:15.638 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:15.191+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:15.638 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:15.381+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:15.638 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:15.381+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:15.638 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:15.381+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:15.638 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:15.382+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:15.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:15.824+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:15.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:15.824+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:15.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:15.824+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:15.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:15.824+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:16.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:16.165+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:16.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:16.165+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:16.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:16.165+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:16.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:16.165+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:16.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:16.358+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:16.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:16.358+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:16.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:16.358+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:16.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:16.359+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:17.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:16.827+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:17.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:16.827+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:17.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:16.827+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:17.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:16.828+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:17.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:17.194+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:17.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:17.194+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:17.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:17.194+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:17.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:17.195+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:17.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:17.317+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:17.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:17.317+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:17.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:17.317+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:17.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:17.317+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:18.153 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:17.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:18.154 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:17.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:18.154 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:17.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:18.154 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:17.832+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:18.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:18.269+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:18.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:18.269+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:18.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:18.269+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:18.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:18.270+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:18.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:18.152+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:18.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:18.152+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:18.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:18.152+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:18.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:18.153+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:19.141 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:28:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:28:18.717+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14134 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:28:19.141 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:18.849+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:19.141 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:18.849+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:19.141 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:18.849+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:19.141 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:18.849+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:19.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:19.140+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:19.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:19.140+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:19.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:19.140+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:19.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:19.141+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:19.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:19.231+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:19.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:19.231+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:19.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:19.231+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:19.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:19.232+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:20.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:19.896+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:20.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:19.896+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:20.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:19.896+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:20.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:19.896+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:20.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:20.143+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:20.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:20.143+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:20.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:20.143+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:20.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:20.144+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:20.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:20.205+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:20.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:20.205+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:20.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:20.205+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:20.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:20.206+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:21.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:21.116+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:21.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:21.116+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:21.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:21.116+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:21.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:21.117+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:21.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:21.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:21.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:21.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:21.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:21.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:21.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:21.165+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:21.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:20.931+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:21.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:20.931+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:21.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:20.931+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:21.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:20.932+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:22.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:21.891+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:22.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:21.891+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:22.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:21.891+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:22.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:21.892+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:22.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:22.121+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:22.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:22.121+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:22.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:22.121+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:22.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:22.122+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:22.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:22.113+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:22.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:22.113+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:22.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:22.113+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:22.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:22.113+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:23.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:23.065+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:23.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:23.065+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:23.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:23.065+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:23.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:23.065+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:23.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:22.915+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:23.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:22.915+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:23.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:22.915+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:23.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:22.915+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:23.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:23.145+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:23.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:23.145+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:23.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:23.145+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:23.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:23.146+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:23.906 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:28:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:28:23.718+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14186 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:28:24.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:24.062+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:24.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:24.062+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:24.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:24.062+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:24.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:24.063+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:24.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:23.904+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:24.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:23.904+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:24.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:23.904+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:24.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:23.905+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:24.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:24.098+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:24.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:24.098+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:24.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:24.098+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:24.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:24.099+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:25.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:25.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:25.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:25.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:25.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:25.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:25.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:25.083+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:25.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:25.049+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:25.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:25.049+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:25.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:25.049+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:25.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:25.049+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:25.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:24.948+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:25.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:24.948+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:25.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:24.948+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:25.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:24.948+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:26.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:26.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:26.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:26.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:26.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:26.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:26.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:26.068+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:26.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:26.042+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:26.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:26.042+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:26.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:26.042+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:26.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:26.042+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:26.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:25.946+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:26.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:25.946+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:26.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:25.946+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:26.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:25.947+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:27.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:27.074+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:27.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:27.074+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:27.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:27.074+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:27.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:27.075+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:27.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:27.063+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:27.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:27.063+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:27.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:27.063+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:27.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:27.064+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:27.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:26.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:27.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:26.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:27.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:26.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:27.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:26.961+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:28.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:28.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:28.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:28.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:28.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:28.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:28.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:28.048+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:28.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:28.015+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:28.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:28.015+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:28.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:28.015+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:28.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:28.015+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:28.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:27.953+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:28.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:27.953+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:28.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:27.953+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:28.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:27.953+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:29.004 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:28:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:28:28.720+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14239 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:28:29.004 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:28.950+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:29.004 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:28.950+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:29.004 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:28.950+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:29.005 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:28.950+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:29.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:29.049+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:29.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:29.049+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:29.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:29.049+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:29.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:29.050+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:29.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:29.004+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:29.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:29.004+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:29.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:29.004+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:29.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:29.004+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:30.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:29.991+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:30.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:29.991+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:30.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:29.991+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:30.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:29.992+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:30.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:30.063+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:30.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:30.063+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:30.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:30.063+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:30.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:30.064+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:30.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:29.907+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:30.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:29.907+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:30.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:29.907+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:30.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:29.908+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:31.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:31.102+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:31.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:31.102+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:31.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:31.102+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:31.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:31.103+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:31.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:31.007+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:31.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:31.007+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:31.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:31.007+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:31.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:31.009+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:31.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:30.892+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:31.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:30.892+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:31.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:30.892+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:31.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:30.893+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:32.152 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:32.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:32.152 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:32.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:32.152 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:32.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:32.152 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:32.048+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:32.153 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:31.853+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:32.153 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:31.853+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:32.153 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:31.853+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:32.153 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:31.853+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:32.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:32.152+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:32.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:32.152+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:32.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:32.152+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:32.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:32.153+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:33.133 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:33.020+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:33.134 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:33.020+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:33.134 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:33.020+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:33.134 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:33.020+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:33.134 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:32.818+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:33.134 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:32.818+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:33.135 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:32.818+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:33.135 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:32.819+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:33.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:33.133+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:33.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:33.133+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:33.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:33.133+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:33.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:33.134+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:34.031 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:28:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:28:33.722+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14291 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:28:34.032 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:33.789+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:34.032 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:33.789+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:34.032 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:33.789+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:34.032 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:33.790+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:34.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:34.176+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:34.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:34.176+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:34.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:34.176+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:34.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:34.177+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:34.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:34.031+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:34.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:34.031+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:34.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:34.031+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:34.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:34.032+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:35.030 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:34.754+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:35.031 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:34.754+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:35.031 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:34.754+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:35.031 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:34.755+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:35.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:35.171+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:35.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:35.171+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:35.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:35.171+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:35.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:35.172+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:35.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:35.030+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:35.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:35.030+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:35.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:35.030+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:35.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:35.031+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:36.020 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:35.743+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:36.020 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:35.743+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:36.021 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:35.743+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:36.021 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:35.744+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:36.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:36.211+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:36.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:36.211+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:36.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:36.211+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:36.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:36.212+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:36.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:36.020+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:36.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:36.020+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:36.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:36.020+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:36.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:36.021+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:37.041 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:36.720+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:37.042 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:36.720+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:37.042 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:36.720+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:37.042 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:36.721+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:37.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:37.173+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:37.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:37.173+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:37.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:37.173+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:37.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:37.174+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:37.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:37.041+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:37.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:37.041+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:37.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:37.041+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:37.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:37.043+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:38.015 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:37.739+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:38.015 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:37.739+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:38.016 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:37.739+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:38.016 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:37.740+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:38.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:38.015+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:38.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:38.015+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:38.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:38.015+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:38.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:38.016+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:38.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:38.150+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:38.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:38.150+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:38.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:38.150+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:38.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:38.151+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:38.985 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:28:38.723+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14343 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:28:38.985 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:38.778+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:38.985 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:38.778+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:38.985 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:38.778+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:38.985 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:38.779+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:39.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:38.984+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:39.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:38.984+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:39.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:38.984+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:39.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:38.985+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:39.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:39.137+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:39.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:39.137+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:39.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:39.137+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:39.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:39.138+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:40.152 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:39.986+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:40.153 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:39.986+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:40.153 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:39.986+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:40.153 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:39.987+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:40.153 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:39.775+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:40.153 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:39.775+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:40.153 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:39.775+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:40.154 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:39.775+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:40.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:40.152+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:40.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:40.152+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:40.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:40.152+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:40.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:40.153+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:40.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:40.773+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:40.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:40.773+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:40.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:40.773+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:40.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:40.774+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:41.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:40.986+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:41.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:40.986+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:41.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:40.986+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:41.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:40.987+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:41.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:41.139+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:41.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:41.139+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:41.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:41.139+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:41.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:41.140+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:41.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:41.812+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:41.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:41.812+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:41.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:41.812+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:41.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:41.813+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:42.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:42.167+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:42.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:42.167+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:42.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:42.167+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:42.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:42.168+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:42.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:41.969+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:42.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:41.969+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:42.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:41.969+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:42.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:41.970+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:43.144 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:42.989+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:43.144 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:42.989+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:43.144 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:42.989+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:43.144 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:42.990+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:43.145 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:42.805+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:43.145 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:42.805+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:43.145 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:42.805+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:43.145 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:42.806+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:43.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:43.144+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:43.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:43.144+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:43.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:43.144+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:43.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:43.145+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:43.800 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:28:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:28:43.725+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14395 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:28:44.144 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:43.800+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:44.144 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:43.800+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:44.144 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:43.800+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:44.144 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:43.801+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:44.144 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:44.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:44.144 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:44.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:44.145 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:44.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:44.145 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:44.023+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:44.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:44.143+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:44.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:44.143+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:44.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:44.143+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:44.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:44.144+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:45.043 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:44.786+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:45.043 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:44.786+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:45.043 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:44.786+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:45.043 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:44.787+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:45.422 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:45.042+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:45.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:45.042+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:45.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:45.042+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:45.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:45.043+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:45.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:45.178+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:45.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:45.178+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:45.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:45.178+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:45.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:45.179+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:45.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:45.822+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:45.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:45.822+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:45.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:45.823+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:45.923 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:45.823+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:46.357 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:46.053+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:46.358 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:46.053+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:46.358 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:46.053+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:46.358 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:46.054+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:46.358 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:46.219+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:46.358 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:46.219+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:46.358 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:46.219+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:46.358 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:46.220+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:47.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:47.044+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:47.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:47.044+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:47.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:47.044+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:47.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:47.045+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:47.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:46.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:47.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:46.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:47.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:46.832+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:47.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:46.833+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:47.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:47.264+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:47.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:47.264+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:47.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:47.264+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:47.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:47.264+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:48.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:48.081+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:48.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:48.081+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:48.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:48.081+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:48.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:48.082+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:48.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:47.847+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:48.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:47.847+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:48.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:47.847+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:48.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:47.848+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:48.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:48.312+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:48.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:48.312+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:48.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:48.312+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:48.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:48.313+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:49.104 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:28:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:28:48.726+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14448 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:28:49.104 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:48.897+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:49.104 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:48.897+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:49.105 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:48.897+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:49.105 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:48.898+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:49.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:49.349+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:49.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:49.349+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:49.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:49.349+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:49.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:49.350+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:49.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:49.104+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:49.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:49.104+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:49.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:49.104+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:49.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:49.105+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:50.130 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:49.859+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:50.131 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:49.859+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:50.131 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:49.859+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:50.131 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:49.860+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:50.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:50.318+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:50.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:50.318+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:50.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:50.318+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:50.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:50.319+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:50.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:50.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:50.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:50.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:50.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:50.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:50.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:50.131+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:51.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:50.882+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:51.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:50.882+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:51.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:50.883+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:51.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:50.883+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:51.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:51.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:51.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:51.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:51.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:51.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:51.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:51.118+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:51.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:51.279+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:51.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:51.279+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:51.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:51.279+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:51.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:51.280+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:52.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:52.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:52.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:52.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:52.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:52.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:52.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:52.103+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:52.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:51.873+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:52.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:51.873+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:52.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:51.873+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:52.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:51.873+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:52.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:52.247+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:52.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:52.247+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:52.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:52.247+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:52.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:52.248+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:53.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:53.060+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:53.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:53.061+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:53.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:53.061+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:53.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:53.061+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:53.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:52.915+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:53.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:52.915+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:53.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:52.915+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:53.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:52.916+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:53.616 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:53.222+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:53.616 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:53.222+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:53.616 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:53.222+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:53.616 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:53.223+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:53.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:28:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:28:53.728+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14500 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:28:54.254 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:53.963+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:54.254 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:53.963+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:54.255 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:53.963+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:54.255 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:53.964+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:54.255 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:54.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:54.255 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:54.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:54.255 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:54.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:54.255 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:54.084+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:54.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:54.254+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:54.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:54.254+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:54.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:54.254+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:54.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:54.255+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:55.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:55.211+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:55.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:55.211+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:55.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:55.211+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:55.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:55.212+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:55.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:55.096+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:55.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:55.096+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:55.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:55.096+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:55.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:55.097+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:55.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:55.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:55.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:55.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:55.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:55.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:55.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:55.003+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:56.422 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:56.198+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:56.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:56.198+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:56.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:56.198+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:56.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:56.198+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:56.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:56.071+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:56.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:56.071+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:56.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:56.071+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:56.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:56.072+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:56.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:55.959+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:56.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:55.959+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:56.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:55.959+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:56.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:55.959+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:57.229 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:57.028+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:57.230 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:57.028+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:57.230 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:57.028+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:57.230 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:57.028+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:57.230 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:56.949+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:57.230 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:56.949+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:57.230 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:56.949+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:57.231 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:56.950+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:57.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:57.229+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:57.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:57.229+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:57.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:57.229+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:57.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:57.230+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:58.265 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:58.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:58.265 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:58.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:58.265 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:58.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:58.266 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:58.052+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:58.266 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:57.965+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:58.266 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:57.965+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:58.266 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:57.965+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:58.266 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:57.965+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:58.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:58.265+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:58.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:58.265+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:58.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:58.265+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:58.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:58.265+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:28:59.004 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:28:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:28:58.729+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14552 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:28:59.005 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:58.966+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:59.005 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:58.966+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:59.005 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:58.966+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:28:59.005 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:58.966+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:59.313 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:59.004+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:59.313 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:59.004+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:59.313 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:59.004+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:28:59.313 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:28:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:28:59.005+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:28:59.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:59.313+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:59.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:59.313+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:59.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:59.313+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:28:59.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:28:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:28:59.314+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:00.290 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:00.016+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:00.290 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:00.016+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:00.290 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:00.016+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:00.291 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:00.017+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:00.291 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:59.963+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:00.291 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:59.963+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:00.291 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:59.963+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:00.291 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:28:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:28:59.964+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:00.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:00.289+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:00.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:00.289+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:00.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:00.289+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:00.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:00.290+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:01.329 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:01.025+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:01.329 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:01.025+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:01.329 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:01.025+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:01.329 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:01.026+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:01.330 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:00.983+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:01.330 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:00.983+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:01.330 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:00.983+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:01.330 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:00.984+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:01.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:01.328+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:01.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:01.328+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:01.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:01.328+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:01.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:01.329+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:02.351 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:02.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:02.351 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:02.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:02.351 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:02.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:02.352 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:02.023+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:02.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:01.944+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:02.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:01.944+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:02.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:01.944+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:02.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:01.945+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:02.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:02.351+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:02.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:02.351+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:02.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:02.351+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:02.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:02.352+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:03.282 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:02.987+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:03.283 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:02.987+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:03.283 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:02.987+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:03.283 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:02.988+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:03.283 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:02.979+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:03.283 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:02.979+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:03.283 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:02.979+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:03.284 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:02.980+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:03.617 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:03.346+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:03.617 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:03.346+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:03.618 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:03.346+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:03.618 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:03.347+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:03.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:29:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:29:03.731+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14604 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:29:04.299 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:03.951+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:04.300 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:03.951+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:04.300 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:03.951+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:04.300 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:03.952+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:04.300 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:04.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:04.300 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:04.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:04.301 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:04.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:04.301 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:04.007+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:04.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:04.299+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:04.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:04.299+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:04.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:04.299+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:04.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:04.300+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:05.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:04.941+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:05.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:04.941+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:05.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:04.941+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:05.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:04.942+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:05.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:05.012+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:05.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:05.012+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:05.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:05.012+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:05.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:05.013+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:05.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:05.290+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:05.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:05.290+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:05.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:05.290+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:05.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:05.291+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:06.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:05.909+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:06.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:05.909+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:06.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:05.909+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:06.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:05.910+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:06.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:05.974+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:06.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:05.974+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:06.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:05.974+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:06.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:05.975+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:06.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:06.323+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:06.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:06.323+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:06.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:06.323+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:06.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:06.323+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:07.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:06.867+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:07.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:06.867+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:07.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:06.867+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:07.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:06.868+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:07.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:06.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:07.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:06.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:07.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:06.992+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:07.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:06.993+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:07.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:07.373+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:07.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:07.373+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:07.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:07.373+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:07.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:07.374+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:08.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:07.851+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:08.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:07.851+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:08.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:07.851+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:08.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:07.852+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:08.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:07.995+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:08.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:07.995+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:08.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:07.995+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:08.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:07.996+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:08.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:08.405+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:08.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:08.405+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:08.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:08.405+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:08.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:08.406+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:08.998 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:29:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:29:08.732+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14661 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:29:08.998 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:08.848+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:08.998 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:08.848+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:08.998 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:08.848+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:08.998 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:08.850+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:09.386 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:08.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:09.386 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:08.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:09.386 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:08.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:09.386 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:08.998+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:09.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:09.385+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:09.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:09.385+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:09.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:09.385+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:09.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:09.386+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:10.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:09.953+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:10.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:09.953+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:10.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:09.953+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:10.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:09.954+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:10.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:09.886+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:10.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:09.886+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:10.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:09.886+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:10.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:09.888+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:10.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:10.372+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:10.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:10.372+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:10.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:10.372+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:10.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:10.373+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:11.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:10.956+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:11.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:10.956+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:11.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:10.956+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:11.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:10.957+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:11.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:10.869+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:11.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:10.869+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:11.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:10.869+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:11.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:10.870+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:11.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:11.415+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:11.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:11.415+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:11.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:11.415+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:11.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:11.416+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:12.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:12.004+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:12.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:12.004+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:12.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:12.004+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:12.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:12.004+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:12.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:11.852+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:12.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:11.852+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:12.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:11.852+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:12.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:11.854+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:12.837 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:12.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:12.838 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:12.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:12.838 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:12.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:12.838 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:12.465+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:13.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:12.837+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:13.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:12.837+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:13.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:12.837+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:13.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:12.838+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:13.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:13.012+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:13.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:13.012+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:13.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:13.012+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:13.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:13.012+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:13.728 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:13.478+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:13.729 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:13.478+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:13.729 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:13.478+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:13.729 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:13.479+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:14.005 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:29:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:29:13.734+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14713 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:29:14.006 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:13.809+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:14.006 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:13.809+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:14.006 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:13.809+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:14.006 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:13.811+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:14.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:14.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:14.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:14.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:14.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:14.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:14.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:14.008+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:14.651 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:14.495+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:14.652 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:14.495+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:14.652 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:14.495+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:14.652 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:14.497+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:14.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:14.806+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:14.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:14.806+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:14.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:14.806+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:14.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:14.807+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:15.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:14.980+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:15.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:14.980+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:15.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:14.980+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:15.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:14.981+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:15.777 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:15.479+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:15.777 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:15.479+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:15.777 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:15.479+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:15.777 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:15.481+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:16.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:15.777+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:16.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:15.777+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:16.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:15.777+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:16.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:15.778+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:16.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:15.988+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:16.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:15.988+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:16.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:15.988+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:16.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:15.988+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:16.788 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:16.485+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:16.788 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:16.485+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:16.788 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:16.485+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:16.788 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:16.486+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:17.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:16.941+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:17.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:16.941+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:17.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:16.941+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:17.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:16.941+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:17.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:16.787+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:17.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:16.787+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:17.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:16.787+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:17.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:16.788+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:17.760 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:17.475+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:17.760 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:17.475+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:17.760 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:17.475+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:17.761 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:17.476+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:18.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:17.760+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:18.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:17.760+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:18.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:17.760+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:18.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:17.761+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:18.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:17.927+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:18.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:17.927+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:18.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:17.927+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:18.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:17.928+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:18.785 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:29:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:29:18.735+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14766 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:29:18.785 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:18.526+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:18.785 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:18.526+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:18.785 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:18.526+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:18.785 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:18.526+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:19.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:18.934+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:19.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:18.934+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:19.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:18.934+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:19.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:18.934+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:19.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:18.784+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:19.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:18.784+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:19.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:18.784+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:19.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:18.785+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:19.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:19.534+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:19.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:19.534+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:19.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:19.534+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:19.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:19.535+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:19.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:19.765+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:19.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:19.765+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:19.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:19.765+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:19.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:19.766+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:20.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:19.968+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:20.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:19.968+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:20.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:19.968+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:20.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:19.969+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:20.848 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:20.496+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:20.848 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:20.496+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:20.848 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:20.496+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:20.848 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:20.497+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:20.848 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:20.739+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:20.849 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:20.739+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:20.849 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:20.739+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:20.849 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:20.740+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:21.172 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:21.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:21.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:21.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:21.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:21.007+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:21.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:21.008+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:21.745 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:21.481+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:21.745 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:21.481+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:21.746 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:21.481+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:21.746 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:21.482+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:22.044 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:21.745+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:22.044 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:21.745+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:22.044 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:21.745+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:22.045 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:21.746+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:22.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:22.044+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:22.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:22.044+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:22.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:22.044+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:22.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:22.045+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:22.698 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:22.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:22.699 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:22.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:22.699 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:22.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:22.699 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:22.448+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:23.048 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:22.698+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:23.048 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:22.698+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:23.048 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:22.698+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:23.048 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:22.699+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:23.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:23.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:23.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:23.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:23.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:23.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:23.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:23.048+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:23.738 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:23.455+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:23.738 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:23.455+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:23.738 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:23.455+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:23.738 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:23.456+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:24.069 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:29:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:29:23.737+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14818 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:29:24.070 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:23.737+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:24.070 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:23.737+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:24.070 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:23.737+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:24.070 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:23.738+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:24.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:24.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:24.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:24.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:24.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:24.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:24.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:24.069+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:24.774 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:24.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:24.774 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:24.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:24.775 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:24.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:24.775 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:24.465+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:25.079 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:24.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:25.079 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:24.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:25.079 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:24.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:25.079 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:24.775+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:25.343 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:25.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:25.343 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:25.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:25.344 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:25.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:25.344 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:25.079+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:25.784 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:25.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:25.784 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:25.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:25.784 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:25.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:25.785 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:25.511+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:26.125 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:25.784+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:26.125 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:25.784+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:26.125 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:25.784+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:26.126 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:25.785+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:26.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:26.125+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:26.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:26.125+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:26.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:26.125+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:26.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:26.125+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:26.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:26.748+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:26.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:26.748+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:26.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:26.748+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:26.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:26.748+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:26.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:26.513+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:26.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:26.513+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:26.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:26.513+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:26.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:26.513+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:27.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:27.163+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:27.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:27.163+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:27.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:27.163+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:27.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:27.163+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:27.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:27.756+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:27.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:27.756+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:27.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:27.756+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:27.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:27.757+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:27.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:27.545+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:27.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:27.545+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:27.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:27.545+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:27.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:27.546+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:28.566 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:28.181+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:28.566 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:28.181+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:28.567 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:28.181+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:28.567 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:28.182+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:28.922 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:29:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:29:28.738+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14871 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:29:28.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:28.566+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:28.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:28.566+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:28.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:28.566+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:28.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:28.567+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:28.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:28.763+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:28.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:28.763+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:28.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:28.763+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:28.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:28.764+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:29.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:29.182+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:29.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:29.182+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:29.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:29.182+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:29.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:29.183+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:29.922 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:29.748+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:29.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:29.748+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:29.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:29.748+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:29.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:29.749+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:29.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:29.570+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:29.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:29.570+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:29.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:29.570+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:29.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:29.572+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:30.561 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:30.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:30.562 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:30.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:30.562 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:30.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:30.562 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:30.218+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:30.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:30.560+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:30.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:30.560+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:30.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:30.560+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:30.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:30.561+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:30.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:30.754+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:30.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:30.754+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:30.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:30.754+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:30.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:30.755+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:31.570 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:31.231+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:31.570 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:31.231+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:31.570 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:31.231+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:31.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:31.232+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:31.922 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:31.575+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:31.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:31.575+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:31.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:31.575+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:31.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:31.577+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:31.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:31.763+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:31.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:31.763+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:31.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:31.763+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:31.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:31.763+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:32.585 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:32.191+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:32.586 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:32.191+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:32.586 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:32.191+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:32.586 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:32.191+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:32.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:32.585+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:32.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:32.585+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:32.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:32.585+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:32.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:32.586+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:32.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:32.744+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:32.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:32.744+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:32.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:32.744+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:32.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:32.745+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:33.611 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:33.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:33.611 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:33.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:33.611 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:33.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:33.611 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:33.203+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:33.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:29:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:29:33.740+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14922 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:29:33.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:33.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:33.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:33.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:33.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:33.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:33.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:33.611+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:33.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:33.739+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:33.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:33.739+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:33.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:33.739+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:33.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:33.740+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:34.626 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:34.194+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:34.626 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:34.194+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:34.626 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:34.194+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:34.626 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:34.194+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:34.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:34.742+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:34.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:34.742+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:34.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:34.742+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:34.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:34.743+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:34.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:34.625+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:34.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:34.625+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:34.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:34.625+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:34.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:34.626+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:35.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:35.163+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:35.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:35.163+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:35.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:35.163+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:35.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:35.164+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:35.774 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:35.639+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:35.775 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:35.639+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:35.775 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:35.639+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:35.775 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:35.641+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:36.119 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:35.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:36.119 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:35.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:36.119 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:35.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:36.119 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:35.775+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:36.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:36.118+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:36.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:36.118+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:36.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:36.118+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:36.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:36.119+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:36.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:36.750+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:36.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:36.750+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:36.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:36.750+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:36.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:36.751+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:36.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:36.636+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:36.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:36.636+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:36.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:36.636+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:36.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:36.637+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:37.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:37.114+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:37.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:37.114+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:37.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:37.114+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:37.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:37.114+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:37.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:37.652+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:37.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:37.652+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:37.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:37.652+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:37.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:37.653+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:37.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:37.702+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:37.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:37.702+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:37.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:37.702+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:37.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:37.703+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:38.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:38.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:38.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:38.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:38.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:38.076+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:38.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:38.076+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:38.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:29:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:29:38.741+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 14975 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:29:38.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:38.679+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:38.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:38.679+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:38.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:38.679+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:38.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:38.681+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:38.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:38.670+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:38.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:38.670+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:38.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:38.670+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:38.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:38.670+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:39.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:39.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:39.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:39.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:39.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:39.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:39.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:39.064+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:40.062 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:39.689+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:40.062 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:39.689+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:40.063 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:39.689+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:40.063 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:39.690+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:40.063 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:39.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:40.063 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:39.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:40.063 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:39.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:40.063 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:39.701+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:40.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:40.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:40.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:40.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:40.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:40.062+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:40.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:40.063+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:40.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:40.718+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:40.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:40.718+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:40.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:40.718+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:40.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:40.719+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:40.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:40.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:40.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:40.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:40.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:40.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:40.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:40.669+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:41.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:41.080+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:41.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:41.080+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:41.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:41.080+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:41.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:41.081+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:41.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:41.761+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:41.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:41.761+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:41.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:41.761+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:41.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:41.762+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:41.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:41.641+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:41.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:41.641+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:41.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:41.641+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:41.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:41.642+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:42.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:42.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:42.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:42.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:42.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:42.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:42.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:42.075+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:42.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:42.773+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:42.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:42.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:42.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:42.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:42.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:42.774+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:42.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:42.669+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:42.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:42.669+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:42.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:42.669+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:42.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:42.670+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:43.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:43.053+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:43.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:43.053+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:43.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:43.053+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:43.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:43.053+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:43.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:29:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:29:43.743+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15027 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:29:43.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:43.750+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:43.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:43.750+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:43.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:43.750+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:43.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:43.751+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:43.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:43.673+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:43.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:43.673+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:43.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:43.673+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:43.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:43.674+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:44.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:44.028+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:44.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:44.028+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:44.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:44.028+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:44.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:44.028+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:45.026 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:44.741+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:45.026 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:44.741+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:45.026 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:44.741+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:45.026 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:44.742+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:45.027 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:44.694+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:45.027 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:44.694+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:45.027 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:44.694+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:45.027 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:44.695+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:45.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:45.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:45.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:45.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:45.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:45.025+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:45.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:45.026+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:46.007 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:45.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:46.007 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:45.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:46.007 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:45.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:46.007 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:45.717+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:46.007 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:45.786+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:46.008 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:45.786+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:46.008 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:45.786+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:46.008 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:45.787+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:46.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:46.006+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:46.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:46.006+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:46.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:46.006+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:46.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:46.007+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:47.050 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:46.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:47.051 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:46.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:47.051 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:46.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:47.051 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:46.708+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:47.051 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:46.800+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:47.051 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:46.800+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:47.052 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:46.800+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:47.052 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:46.800+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:47.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:47.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:47.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:47.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:47.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:47.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:47.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:47.050+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:48.013 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:47.777+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:48.014 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:47.777+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:48.014 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:47.777+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:48.014 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:47.778+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:48.014 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:47.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:48.014 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:47.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:48.015 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:47.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:48.015 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:47.747+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:48.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:48.012+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:48.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:48.012+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:48.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:48.012+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:48.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:48.013+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:49.058 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:29:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:29:48.743+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15080 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:29:49.058 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:48.784+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:49.058 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:48.784+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:49.059 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:48.784+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:49.059 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:48.785+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:49.059 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:48.770+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:49.059 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:48.770+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:49.059 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:48.770+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:49.059 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:48.771+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:49.422 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:49.056+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:49.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:49.056+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:49.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:49.056+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:49.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:49.057+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:50.055 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:49.736+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:50.055 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:49.736+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:50.056 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:49.736+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:50.056 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:49.737+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:50.056 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:49.812+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:50.056 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:49.812+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:50.056 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:49.812+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:50.056 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:49.813+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:50.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:50.053+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:50.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:50.054+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:50.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:50.054+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:50.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:50.054+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:51.077 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:50.735+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:51.077 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:50.735+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:51.077 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:50.735+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:51.077 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:50.736+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:51.078 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:50.802+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:51.078 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:50.802+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:51.078 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:50.802+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:51.078 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:50.803+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:51.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:51.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:51.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:51.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:51.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:51.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:51.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:51.078+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:51.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:51.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:51.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:51.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:51.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:51.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:51.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:51.688+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:51.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:51.799+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:51.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:51.799+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:51.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:51.799+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:51.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:51.800+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:52.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:52.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:52.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:52.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:52.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:52.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:52.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:52.080+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:53.058 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:52.801+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:53.058 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:52.801+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:53.058 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:52.801+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:53.059 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:52.802+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:53.059 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:52.677+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:53.059 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:52.677+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:53.059 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:52.677+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:53.059 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:52.678+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:53.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:53.056+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:53.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:53.056+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:53.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:53.056+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:53.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:53.057+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:53.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:29:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:29:53.745+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15132 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:29:53.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:53.770+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:53.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:53.770+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:53.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:53.770+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:53.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:53.770+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:53.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:53.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:53.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:53.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:53.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:53.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:53.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:53.649+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:54.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:54.053+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:54.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:54.053+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:54.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:54.053+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:54.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:54.053+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:54.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:54.726+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:54.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:54.726+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:54.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:54.726+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:54.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:54.727+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:54.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:54.615+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:54.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:54.615+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:54.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:54.615+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:54.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:54.615+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:55.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:55.100+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:55.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:55.100+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:55.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:55.100+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:55.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:55.101+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:55.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:55.645+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:55.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:55.645+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:55.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:55.645+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:55.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:55.647+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:55.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:55.697+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:55.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:55.697+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:55.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:55.697+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:55.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:55.699+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:56.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:56.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:56.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:56.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:56.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:56.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:56.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:56.078+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:56.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:56.748+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:56.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:56.748+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:56.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:56.748+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:56.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:56.748+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:56.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:56.655+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:56.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:56.655+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:56.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:56.655+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:56.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:56.656+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:57.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:57.040+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:57.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:57.040+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:57.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:57.040+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:57.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:57.041+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:57.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:57.734+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:57.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:57.734+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:57.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:57.734+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:57.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:57.735+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:57.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:57.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:57.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:57.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:57.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:57.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:57.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:57.627+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:58.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:58.065+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:58.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:58.065+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:58.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:58.065+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:58.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:58.066+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:58.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:29:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:29:58.746+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15185 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:29:58.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:58.731+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:58.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:58.731+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:58.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:58.731+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:58.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:58.733+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:58.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:58.614+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:58.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:58.614+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:58.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:58.614+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:58.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:58.616+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:29:59.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:59.067+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:59.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:59.067+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:59.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:59.067+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:29:59.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:29:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:29:59.068+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:59.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:59.723+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:59.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:59.723+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:59.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:59.723+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:29:59.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:29:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:29:59.724+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:29:59.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:59.593+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:59.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:59.593+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:59.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:59.593+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:29:59.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:29:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:29:59.593+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:00.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:00.095+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:00.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:00.095+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:00.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:00.095+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:00.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:00.095+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:00.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:00.698+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:00.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:00.698+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:00.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:00.698+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:00.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:00.700+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:00.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:00.606+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:00.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:00.606+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:00.606+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:00.606+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:01.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:01.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:01.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:01.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:01.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:01.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:01.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:01.064+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:01.733 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:01.654+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:01.733 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:01.654+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:01.733 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:01.654+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:01.733 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:01.655+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:02.075 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:01.732+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:02.075 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:01.732+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:02.075 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:01.732+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:02.075 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:01.733+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:02.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:02.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:02.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:02.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:02.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:02.074+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:02.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:02.075+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:02.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:02.616+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:02.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:02.616+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:02.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:02.616+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:02.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:02.617+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:02.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:02.714+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:02.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:02.714+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:02.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:02.714+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:02.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:02.715+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:03.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:03.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:03.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:03.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:03.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:03.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:03.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:03.047+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:03.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:30:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:30:03.749+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15237 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:30:03.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:03.576+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:03.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:03.576+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:03.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:03.576+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:03.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:03.577+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:03.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:03.711+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:03.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:03.711+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:03.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:03.711+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:03.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:03.712+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:04.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:04.060+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:04.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:04.060+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:04.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:04.060+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:04.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:04.060+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:04.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:04.531+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:04.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:04.531+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:04.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:04.531+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:04.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:04.533+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:04.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:04.750+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:04.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:04.750+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:04.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:04.750+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:04.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:04.751+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:05.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:05.032+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:05.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:05.032+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:05.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:05.032+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:05.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:05.032+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:05.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:05.560+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:05.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:05.560+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:05.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:05.560+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:05.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:05.561+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:05.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:05.703+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:05.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:05.703+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:05.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:05.703+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:05.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:05.704+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:06.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:06.015+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:06.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:06.015+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:06.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:06.015+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:06.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:06.016+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:06.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:06.688+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:06.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:06.688+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:06.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:06.688+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:06.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:06.689+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:06.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:06.540+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:06.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:06.540+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:06.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:06.540+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:06.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:06.541+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:07.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:07.019+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:07.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:07.019+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:07.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:07.019+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:07.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:07.020+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:07.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:07.499+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:07.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:07.499+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:07.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:07.499+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:07.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:07.500+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:07.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:07.655+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:07.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:07.655+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:07.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:07.656+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:07.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:07.656+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:08.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:07.976+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:08.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:07.976+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:08.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:07.976+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:08.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:07.977+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:08.751 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:08.460+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:08.751 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:08.460+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:08.752 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:08.460+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:08.752 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:08.461+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:08.752 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:08.618+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:08.752 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:08.618+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:08.752 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:08.618+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:08.752 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:08.618+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:09.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:30:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:30:08.750+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15290 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:30:09.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:08.938+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:09.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:08.938+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:09.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:08.938+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:09.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:08.939+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:09.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:09.599+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:09.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:09.599+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:09.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:09.599+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:09.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:09.599+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:09.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:09.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:09.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:09.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:09.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:09.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:09.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:09.464+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:10.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:09.949+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:10.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:09.949+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:10.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:09.949+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:10.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:09.950+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:10.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:10.603+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:10.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:10.603+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:10.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:10.603+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:10.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:10.603+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:10.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:10.452+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:10.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:10.452+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:10.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:10.452+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:10.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:10.453+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:11.232 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:10.939+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:11.232 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:10.939+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:11.232 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:10.939+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:11.233 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:10.940+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:11.559 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:11.421+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:11.560 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:11.421+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:11.560 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:11.421+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:11.560 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:11.421+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:11.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:11.559+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:11.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:11.559+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:11.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:11.559+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:11.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:11.560+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:12.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:11.943+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:12.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:11.943+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:12.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:11.943+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:12.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:11.944+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:12.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:12.515+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:12.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:12.515+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:12.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:12.515+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:12.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:12.516+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:12.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:12.455+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:12.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:12.455+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:12.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:12.455+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:12.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:12.456+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:13.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:12.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:13.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:12.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:13.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:12.986+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:13.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:12.987+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:13.753 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:13.483+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:13.753 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:13.483+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:13.753 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:13.483+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:13.753 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:13.484+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:13.753 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:13.527+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:13.754 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:13.527+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:13.754 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:13.527+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:13.754 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:13.528+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:14.026 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:30:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:30:13.752+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15342 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:30:14.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:14.026+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:14.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:14.026+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:14.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:14.026+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:14.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:14.027+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:14.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:14.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:14.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:14.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:14.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:14.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:14.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:14.448+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:14.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:14.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:14.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:14.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:14.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:14.501+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:14.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:14.501+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:15.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:15.046+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:15.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:15.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:15.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:15.047+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:15.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:15.047+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:15.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:15.433+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:15.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:15.433+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:15.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:15.433+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:15.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:15.434+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:15.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:15.551+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:15.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:15.551+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:15.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:15.551+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:15.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:15.553+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:16.386 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:16.080+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:16.386 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:16.080+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:16.386 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:16.080+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:16.387 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:16.081+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:16.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:16.385+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:16.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:16.385+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:16.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:16.385+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:16.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:16.386+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:16.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:16.570+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:16.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:16.570+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:16.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:16.570+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:16.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:16.570+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:17.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:17.112+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:17.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:17.112+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:17.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:17.112+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:17.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:17.113+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:17.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:17.344+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:17.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:17.344+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:17.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:17.344+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:17.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:17.345+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:17.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:17.562+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:17.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:17.562+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:17.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:17.562+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:17.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:17.563+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:18.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:18.303+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:18.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:18.303+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:18.423 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:18.303+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:18.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:18.304+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:18.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:18.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:18.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:18.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:18.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:18.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:18.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:18.073+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:18.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:30:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:30:18.753+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15395 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:30:18.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:18.568+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:18.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:18.568+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:18.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:18.568+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:18.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:18.570+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:19.343 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:19.039+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:19.344 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:19.039+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:19.344 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:19.039+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:19.344 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:19.040+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:19.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:19.343+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:19.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:19.343+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:19.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:19.343+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:19.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:19.344+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:19.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:19.545+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:19.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:19.545+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:19.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:19.545+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:19.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:19.546+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:20.387 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:20.084+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:20.387 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:20.084+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:20.387 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:20.084+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:20.387 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:20.085+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:20.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:20.386+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:20.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:20.386+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:20.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:20.386+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:20.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:20.387+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:20.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:20.511+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:20.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:20.511+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:20.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:20.511+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:20.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:20.512+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:21.314 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:21.055+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:21.315 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:21.055+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:21.315 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:21.055+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:21.315 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:21.055+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:21.570 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:21.356+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:21.571 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:21.356+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:21.571 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:21.356+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:21.571 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:21.356+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:21.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:21.559+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:21.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:21.559+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:21.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:21.559+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:21.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:21.559+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:22.371 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:22.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:22.371 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:22.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:22.372 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:22.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:22.372 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:22.067+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:22.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:22.370+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:22.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:22.370+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:22.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:22.370+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:22.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:22.371+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:22.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:22.547+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:22.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:22.547+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:22.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:22.547+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:22.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:22.548+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:23.374 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:23.088+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:23.374 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:23.088+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:23.375 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:23.088+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:23.375 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:23.089+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:23.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:23.507+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:23.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:23.507+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:23.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:23.507+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:23.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:23.508+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:23.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:23.373+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:23.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:23.373+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:23.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:23.373+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:23.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:23.374+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:24.050 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:30:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:30:23.755+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15447 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:30:24.411 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:24.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:24.411 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:24.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:24.411 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:24.049+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:24.411 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:24.050+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:24.662 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:24.410+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:24.662 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:24.410+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:24.662 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:24.410+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:24.662 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:24.412+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:24.663 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:24.470+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:24.663 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:24.470+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:24.663 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:24.470+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:24.663 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:24.471+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:25.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:25.073+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:25.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:25.073+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:25.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:25.073+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:25.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:25.073+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:25.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:25.461+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:25.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:25.461+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:25.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:25.461+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:25.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:25.462+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:25.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:25.519+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:25.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:25.519+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:25.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:25.519+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:25.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:25.519+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:26.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:26.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:26.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:26.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:26.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:26.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:26.353 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:26.051+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:26.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:26.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:26.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:26.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:26.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:26.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:26.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:26.465+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:26.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:26.568+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:26.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:26.568+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:26.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:26.568+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:26.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:26.568+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:27.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:27.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:27.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:27.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:27.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:27.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:27.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:27.018+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:27.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:27.506+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:27.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:27.506+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:27.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:27.506+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:27.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:27.507+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:27.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:27.548+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:27.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:27.548+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:27.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:27.548+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:27.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:27.549+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:28.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:28.008+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:28.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:28.008+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:28.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:28.008+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:28.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:28.009+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:28.757 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:28.561+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:28.758 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:28.561+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:28.758 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:28.561+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:28.758 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:28.562+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:28.758 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:28.480+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:28.759 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:28.480+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:28.759 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:28.480+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:28.759 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:28.481+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:29.009 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:30:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:30:28.757+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15500 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:30:29.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:29.009+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:29.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:29.009+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:29.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:29.009+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:29.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:29.009+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:29.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:29.593+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:29.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:29.593+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:29.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:29.593+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:29.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:29.595+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:29.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:29.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:29.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:29.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:29.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:29.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:29.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:29.517+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:30.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:30.010+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:30.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:30.010+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:30.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:30.010+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:30.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:30.011+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:30.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:30.644+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:30.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:30.644+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:30.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:30.644+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:30.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:30.645+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:30.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:30.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:30.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:30.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:30.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:30.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:30.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:30.517+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:31.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:31.003+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:31.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:31.003+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:31.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:31.003+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:31.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:31.004+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:31.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:31.606+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:31.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:31.606+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:31.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:31.606+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:31.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:31.608+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:31.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:31.505+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:31.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:31.505+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:31.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:31.505+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:31.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:31.506+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:32.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:32.028+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:32.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:32.028+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:32.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:32.028+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:32.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:32.029+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:32.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:32.560+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:32.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:32.560+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:32.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:32.560+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:32.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:32.561+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:32.550+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:32.550+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:32.550+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:32.551+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:33.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:33.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:33.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:33.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:33.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:33.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:33.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:33.070+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:33.877 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:30:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:30:33.758+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15552 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:30:33.878 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:33.575+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:33.878 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:33.575+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:33.878 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:33.575+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:33.878 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:33.576+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:33.879 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:33.571+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:33.879 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:33.571+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:33.879 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:33.571+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:33.879 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:33.573+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:34.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:34.088+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:34.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:34.088+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:34.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:34.088+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:34.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:34.089+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:34.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:34.607+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:34.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:34.607+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:34.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:34.607+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:34.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:34.608+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:34.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:34.563+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:34.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:34.563+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:34.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:34.563+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:34.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:34.564+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:35.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:35.048+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:35.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:35.048+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:35.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:35.048+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:35.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:35.048+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:35.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:35.621+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:35.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:35.621+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:35.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:35.621+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:35.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:35.623+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:35.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:35.537+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:35.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:35.537+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:35.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:35.537+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:35.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:35.537+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:36.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:36.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:36.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:36.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:36.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:36.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:36.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:36.081+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:36.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:36.616+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:36.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:36.616+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:36.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:36.616+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:36.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:36.617+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:36.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:36.509+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:36.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:36.509+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:36.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:36.509+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:36.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:36.510+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:37.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:37.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:37.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:37.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:37.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:37.078+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:37.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:37.078+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:37.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:37.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:37.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:37.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:37.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:37.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:37.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:37.650+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:37.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:37.537+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:37.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:37.537+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:37.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:37.537+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:37.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:37.538+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:38.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:38.103+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:38.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:38.103+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:38.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:38.103+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:38.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:38.104+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:38.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:30:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:30:38.760+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15604 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:30:38.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:38.525+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:38.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:38.525+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:38.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:38.525+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:38.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:38.526+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:38.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:38.694+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:38.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:38.694+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:38.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:38.694+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:38.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:38.695+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:39.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:39.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:39.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:39.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:39.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:39.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:39.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:39.060+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:39.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:39.531+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:39.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:39.531+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:39.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:39.531+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:39.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:39.532+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:39.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:39.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:39.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:39.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:39.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:39.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:39.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:39.699+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:40.321 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:40.024+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:40.321 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:40.024+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:40.322 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:40.024+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:40.322 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:40.025+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:40.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:40.485+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:40.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:40.485+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:40.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:40.485+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:40.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:40.486+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:41.067 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:40.724+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:41.067 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:40.724+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:41.068 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:40.724+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:41.068 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:40.725+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:41.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:41.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:41.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:41.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:41.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:41.066+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:41.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:41.067+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:41.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:41.486+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:41.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:41.486+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:41.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:41.486+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:41.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:41.487+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:41.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:41.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:41.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:41.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:41.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:41.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:41.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:41.679+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:42.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:42.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:42.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:42.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:42.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:42.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:42.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:42.053+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:42.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:42.488+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:42.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:42.488+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:42.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:42.488+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:42.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:42.489+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:42.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:42.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:42.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:42.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:42.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:42.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:42.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:42.666+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:43.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:43.012+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:43.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:43.012+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:43.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:43.012+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:43.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:43.012+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:43.762 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:43.474+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:43.763 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:43.474+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:43.763 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:43.474+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:43.763 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:43.475+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:43.763 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:43.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:43.763 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:43.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:43.763 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:43.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:43.764 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:43.626+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:44.046 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:30:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:30:43.761+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15656 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:30:44.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:44.046+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:44.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:44.046+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:44.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:44.046+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:44.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:44.046+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:44.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:44.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:44.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:44.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:44.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:44.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:44.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:44.466+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:44.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:44.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:44.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:44.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:44.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:44.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:44.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:44.628+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:45.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:45.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:45.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:45.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:45.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:45.017+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:45.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:45.018+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:45.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:45.472+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:45.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:45.472+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:45.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:45.472+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:45.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:45.473+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:45.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:45.579+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:45.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:45.579+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:45.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:45.579+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:45.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:45.581+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:46.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:46.051+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:46.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:46.051+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:46.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:46.051+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:46.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:46.052+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:46.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:46.505+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:46.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:46.505+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:46.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:46.505+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:46.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:46.505+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:46.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:46.616+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:46.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:46.616+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:46.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:46.616+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:46.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:46.617+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:47.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:47.082+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:47.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:47.082+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:47.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:47.082+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:47.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:47.083+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:47.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:47.551+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:47.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:47.551+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:47.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:47.551+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:47.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:47.551+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:47.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:47.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:47.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:47.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:47.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:47.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:47.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:47.667+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:48.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:48.100+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:48.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:48.100+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:48.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:48.100+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:48.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:48.101+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:48.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:30:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:30:48.763+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15709 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:30:48.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:48.573+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:48.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:48.573+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:48.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:48.573+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:48.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:48.574+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:48.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:48.617+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:48.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:48.617+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:48.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:48.617+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:48.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:48.618+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:49.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:49.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:49.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:49.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:49.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:49.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:49.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:49.059+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:49.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:49.596+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:49.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:49.596+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:49.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:49.596+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:49.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:49.597+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:49.927 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:49.589+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:49.927 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:49.589+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:49.927 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:49.589+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:49.927 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:49.590+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:50.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:50.085+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:50.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:50.085+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:50.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:50.085+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:50.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:50.086+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:50.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:50.572+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:50.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:50.572+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:50.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:50.572+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:50.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:50.573+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:50.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:50.622+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:50.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:50.622+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:50.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:50.622+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:50.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:50.623+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:51.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:51.131+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:51.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:51.131+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:51.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:51.131+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:51.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:51.131+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:51.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:51.604+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:51.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:51.604+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:51.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:51.604+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:51.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:51.605+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:51.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:51.622+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:51.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:51.622+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:51.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:51.622+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:51.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:51.623+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:52.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:52.120+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:52.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:52.120+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:52.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:52.120+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:52.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:52.121+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:52.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:52.655+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:52.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:52.655+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:52.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:52.655+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:52.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:52.657+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:52.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:52.577+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:52.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:52.577+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:52.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:52.577+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:52.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:52.578+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:53.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:53.117+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:53.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:53.117+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:53.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:53.117+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:53.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:53.117+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:53.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:30:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:30:53.765+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15761 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:30:53.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:53.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:53.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:53.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:53.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:53.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:53.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:53.627+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:53.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:53.633+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:53.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:53.633+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:53.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:53.633+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:53.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:53.634+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:54.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:54.166+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:54.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:54.166+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:54.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:54.166+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:54.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:54.167+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:54.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:54.645+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:54.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:54.645+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:54.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:54.645+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:54.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:54.646+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:54.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:54.671+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:54.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:54.671+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:54.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:54.671+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:54.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:54.672+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:55.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:55.163+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:55.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:55.163+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:55.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:55.163+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:55.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:55.163+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:55.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:55.617+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:55.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:55.617+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:55.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:55.617+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:55.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:55.617+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:55.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:55.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:55.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:55.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:55.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:55.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:55.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:55.707+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:56.588 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:56.208+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:56.588 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:56.208+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:56.588 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:56.208+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:56.588 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:56.208+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:56.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:56.594+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:56.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:56.594+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:56.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:56.594+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:56.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:56.595+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:56.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:56.733+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:56.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:56.733+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:56.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:56.733+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:56.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:56.733+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:57.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:57.250+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:57.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:57.250+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:57.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:57.250+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:57.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:57.250+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:57.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:57.630+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:57.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:57.630+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:57.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:57.630+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:57.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:57.631+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:57.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:57.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:57.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:57.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:57.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:57.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:57.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:57.761+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:58.626 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:58.234+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:58.626 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:58.234+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:58.626 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:58.234+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:58.627 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:58.235+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:58.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:30:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:30:58.766+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15814 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:30:58.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:58.719+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:58.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:58.719+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:58.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:58.719+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:58.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:58.720+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:30:58.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:58.625+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:58.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:58.625+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:58.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:58.625+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:58.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:58.626+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:59.611 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:59.202+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:59.611 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:59.202+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:59.611 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:59.202+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:30:59.611 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:30:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:30:59.203+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:59.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:59.610+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:59.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:59.610+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:59.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:59.610+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:30:59.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:30:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:30:59.611+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:30:59.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:59.723+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:59.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:59.723+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:59.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:59.723+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:30:59.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:30:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:30:59.724+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:00.657 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:00.249+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:00.657 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:00.249+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:00.657 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:00.249+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:00.657 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:00.250+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:00.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:00.761+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:00.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:00.761+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:00.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:00.761+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:00.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:00.762+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:00.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:00.656+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:00.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:00.656+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:00.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:00.656+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:00.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:00.657+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:01.570 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:01.267+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:01.570 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:01.267+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:01.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:01.267+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:01.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:01.268+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:01.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:01.670+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:01.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:01.670+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:01.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:01.670+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:01.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:01.671+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:01.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:01.776+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:01.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:01.776+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:01.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:01.776+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:01.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:01.777+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:02.582 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:02.309+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:02.583 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:02.309+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:02.583 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:02.309+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:02.583 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:02.310+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:02.840 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:02.753+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:02.840 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:02.753+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:02.840 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:02.753+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:02.840 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:02.754+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:02.841 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:02.636+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:02.841 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:02.636+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:02.841 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:02.636+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:02.841 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:02.636+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:03.627 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:03.348+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:03.627 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:03.348+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:03.627 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:03.348+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:03.627 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:03.348+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:03.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:31:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:31:03.768+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15866 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:31:03.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:03.681+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:03.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:03.681+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:03.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:03.681+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:03.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:03.682+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:03.752+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:03.752+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:03.752+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:03.753+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:04.598 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:04.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:04.598 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:04.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:04.598 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:04.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:04.599 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:04.308+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:04.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:04.716+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:04.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:04.716+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:04.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:04.716+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:04.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:04.716+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:04.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:04.726+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:04.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:04.726+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:04.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:04.726+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:04.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:04.727+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:05.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:05.298+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:05.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:05.298+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:05.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:05.298+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:05.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:05.299+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:06.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:05.717+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:06.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:05.717+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:06.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:05.717+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:06.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:05.717+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:06.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:05.690+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:06.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:05.690+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:06.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:05.690+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:06.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:05.690+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:06.650 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:06.257+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:06.650 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:06.257+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:06.650 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:06.257+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:06.650 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:06.258+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:06.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:06.675+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:06.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:06.675+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:06.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:06.675+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:06.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:06.676+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:06.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:06.649+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:06.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:06.649+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:06.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:06.649+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:06.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:06.650+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:07.646 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:07.219+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:07.646 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:07.219+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:07.646 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:07.219+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:07.646 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:07.220+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:07.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:07.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:07.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:07.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:07.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:07.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:07.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:07.649+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:07.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:07.645+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:07.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:07.645+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:07.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:07.645+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:07.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:07.646+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:08.602 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:08.199+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:08.603 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:08.199+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:08.603 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:08.199+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:08.603 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:08.199+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:08.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:31:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:31:08.769+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15919 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:31:08.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:08.673+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:08.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:08.673+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:08.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:08.673+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:08.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:08.674+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:08.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:08.602+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:08.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:08.602+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:08.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:08.602+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:08.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:08.602+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:09.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:09.159+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:09.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:09.159+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:09.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:09.159+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:09.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:09.160+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:09.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:09.647+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:09.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:09.647+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:09.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:09.647+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:09.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:09.648+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:09.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:09.659+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:09.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:09.659+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:09.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:09.659+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:09.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:09.660+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:10.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:10.163+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:10.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:10.163+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:10.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:10.163+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:10.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:10.164+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:10.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:10.671+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:10.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:10.671+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:10.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:10.671+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:10.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:10.672+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:10.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:10.643+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:10.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:10.643+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:10.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:10.643+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:10.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:10.644+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:11.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:11.142+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:11.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:11.142+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:11.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:11.142+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:11.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:11.143+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:11.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:11.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:11.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:11.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:11.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:11.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:11.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:11.687+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:11.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:11.603+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:11.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:11.603+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:11.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:11.603+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:11.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:11.603+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:12.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:12.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:12.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:12.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:12.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:12.172+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:12.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:12.173+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:12.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:12.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:12.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:12.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:12.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:12.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:12.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:12.738+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:12.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:12.590+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:12.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:12.590+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:12.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:12.590+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:12.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:12.591+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:13.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:13.126+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:13.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:13.126+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:13.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:13.126+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:13.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:13.127+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:13.884 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:31:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:31:13.771+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 15971 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:31:13.884 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:13.599+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:13.885 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:13.599+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:13.885 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:13.599+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:13.885 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:13.600+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:13.885 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:13.752+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:13.885 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:13.752+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:13.885 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:13.752+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:13.885 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:13.753+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:14.173 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:14.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:14.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:14.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:14.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:14.105+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:14.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:14.106+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:14.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:14.618+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:14.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:14.618+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:14.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:14.618+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:14.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:14.618+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:14.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:14.715+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:14.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:14.715+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:14.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:14.715+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:14.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:14.716+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:15.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:15.146+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:15.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:15.146+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:15.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:15.146+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:15.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:15.147+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:15.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:15.648+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:15.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:15.648+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:15.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:15.648+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:15.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:15.649+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:15.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:15.743+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:15.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:15.743+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:15.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:15.743+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:15.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:15.744+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:16.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:16.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:16.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:16.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:16.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:16.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:16.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:16.163+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:16.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:16.634+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:16.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:16.634+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:16.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:16.634+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:16.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:16.634+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:16.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:16.750+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:16.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:16.750+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:16.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:16.750+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:16.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:16.752+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:17.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:17.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:17.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:17.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:17.632 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:17.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:17.632 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:17.199+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:17.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:17.732+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:17.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:17.732+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:17.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:17.732+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:17.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:17.733+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:17.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:17.630+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:17.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:17.630+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:17.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:17.630+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:17.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:17.631+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:18.601 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:18.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:18.601 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:18.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:18.602 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:18.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:18.602 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:18.218+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:18.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:31:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:31:18.771+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16024 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:31:18.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:18.739+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:18.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:18.739+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:18.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:18.739+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:18.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:18.739+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:18.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:18.599+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:18.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:18.599+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:18.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:18.599+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:18.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:18.601+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:19.591 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:19.261+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:19.592 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:19.261+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:19.592 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:19.261+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:19.592 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:19.262+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:19.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:19.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:19.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:19.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:19.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:19.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:19.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:19.761+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:19.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:19.589+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:19.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:19.589+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:19.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:19.589+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:19.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:19.590+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:20.548 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:20.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:20.549 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:20.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:20.549 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:20.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:20.549 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:20.275+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:20.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:20.729+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:20.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:20.729+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:20.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:20.729+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:20.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:20.730+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:20.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:20.546+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:20.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:20.546+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:20.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:20.546+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:20.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:20.547+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:21.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:21.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:21.570 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:21.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:21.570 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:21.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:21.570 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:21.311+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:21.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:21.567+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:21.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:21.567+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:21.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:21.567+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:21.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:21.568+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:21.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:21.724+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:21.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:21.724+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:21.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:21.724+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:21.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:21.725+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:22.546 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:22.276+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:22.546 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:22.276+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:22.547 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:22.276+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:22.547 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:22.277+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:22.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:22.544+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:22.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:22.544+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:22.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:22.544+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:22.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:22.545+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:22.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:22.683+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:22.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:22.683+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:22.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:22.683+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:22.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:22.684+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:23.526 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:23.247+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:23.527 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:23.247+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:23.527 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:23.247+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:23.527 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:23.248+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:23.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:31:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:31:23.773+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16076 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:31:23.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:23.662+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:23.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:23.662+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:23.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:23.662+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:23.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:23.663+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:23.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:23.525+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:23.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:23.525+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:23.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:23.525+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:23.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:23.526+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:24.662 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:24.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:24.662 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:24.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:24.662 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:24.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:24.662 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:24.274+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:24.663 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:24.477+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:24.663 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:24.477+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:24.663 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:24.477+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:24.663 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:24.477+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:24.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:24.660+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:24.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:24.660+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:24.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:24.660+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:24.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:24.661+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:25.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:25.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:25.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:25.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:25.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:25.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:25.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:25.275+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:25.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:25.483+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:25.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:25.483+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:25.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:25.483+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:25.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:25.484+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:26.102 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:25.673+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:26.102 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:25.673+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:26.102 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:25.673+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:26.103 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:25.674+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:26.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:26.324+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:26.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:26.324+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:26.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:26.324+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:26.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:26.324+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:26.829 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:26.463+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:26.829 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:26.463+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:26.830 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:26.463+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:26.830 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:26.464+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:26.830 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:26.705+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:26.830 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:26.705+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:26.830 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:26.705+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:26.830 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:26.706+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:27.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:27.490+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:27.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:27.490+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:27.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:27.490+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:27.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:27.490+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:27.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:27.343+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:27.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:27.343+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:27.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:27.343+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:27.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:27.344+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:28.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:27.672+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:28.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:27.672+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:28.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:27.672+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:28.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:27.672+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:28.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:28.521+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:28.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:28.521+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:28.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:28.521+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:28.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:28.521+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:28.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:28.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:28.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:28.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:28.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:28.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:28.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:28.389+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:29.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:31:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:31:28.775+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16129 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:31:29.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:28.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:29.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:28.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:29.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:28.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:29.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:28.707+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:29.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:29.570+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:29.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:29.570+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:29.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:29.570+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:29.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:29.571+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:29.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:29.367+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:29.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:29.367+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:29.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:29.367+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:29.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:29.368+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:30.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:29.685+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:30.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:29.685+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:30.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:29.685+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:30.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:29.686+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:30.655 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:30.558+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:30.655 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:30.558+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:30.655 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:30.558+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:30.655 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:30.558+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:30.655 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:30.394+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:30.656 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:30.394+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:30.656 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:30.394+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:30.656 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:30.395+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:30.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:30.654+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:30.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:30.654+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:30.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:30.654+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:30.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:30.654+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:31.669 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:31.574+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:31.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:31.574+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:31.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:31.574+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:31.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:31.575+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:31.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:31.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:31.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:31.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:31.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:31.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:31.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:31.390+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:31.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:31.667+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:31.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:31.667+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:31.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:31.667+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:31.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:31.669+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:32.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:32.540+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:32.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:32.540+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:32.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:32.540+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:32.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:32.541+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:32.667+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:32.667+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:32.667+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:32.668+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:32.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:32.426+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:32.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:32.426+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:32.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:32.426+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:32.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:32.427+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:33.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:33.547+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:33.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:33.547+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:33.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:33.547+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:33.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:33.548+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:33.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:33.392+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:33.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:33.392+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:33.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:33.392+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:33.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:33.393+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:34.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:31:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:31:33.777+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16181 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:31:34.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:33.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:34.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:33.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:34.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:33.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:34.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:33.718+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:34.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:34.419+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:34.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:34.419+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:34.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:34.419+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:34.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:34.419+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:34.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:34.581+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:34.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:34.581+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:34.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:34.581+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:34.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:34.582+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:35.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:34.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:35.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:34.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:35.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:34.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:35.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:34.719+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:35.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:35.387+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:35.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:35.387+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:35.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:35.387+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:35.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:35.388+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:35.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:35.538+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:35.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:35.538+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:35.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:35.538+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:35.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:35.538+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:36.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:35.765+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:36.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:35.765+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:36.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:35.765+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:36.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:35.766+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:36.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:36.360+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:36.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:36.360+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:36.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:36.360+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:36.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:36.360+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:36.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:36.509+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:36.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:36.509+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:36.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:36.509+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:36.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:36.510+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:37.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:36.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:37.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:36.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:37.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:36.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:37.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:36.718+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:37.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:37.508+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:37.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:37.508+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:37.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:37.508+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:37.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:37.509+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:37.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:37.385+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:37.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:37.385+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:37.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:37.385+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:37.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:37.385+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:38.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:37.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:38.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:37.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:38.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:37.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:38.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:37.748+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:38.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:38.484+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:38.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:38.484+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:38.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:38.484+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:38.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:38.485+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:38.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:38.394+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:38.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:38.394+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:38.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:38.394+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:38.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:38.395+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:39.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:31:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:31:38.779+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16234 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:31:39.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:38.786+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:39.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:38.786+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:39.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:38.786+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:39.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:38.787+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:39.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:39.371+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:39.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:39.371+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:39.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:39.371+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:39.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:39.371+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:39.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:39.459+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:39.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:39.459+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:39.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:39.459+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:39.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:39.460+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:40.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:39.807+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:40.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:39.807+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:40.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:39.807+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:40.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:39.808+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:40.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:40.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:40.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:40.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:40.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:40.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:40.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:40.390+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:40.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:40.437+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:40.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:40.437+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:40.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:40.437+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:40.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:40.438+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:41.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:40.858+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:41.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:40.858+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:41.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:40.858+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:41.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:40.859+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:41.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:41.411+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:41.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:41.411+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:41.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:41.411+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:41.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:41.411+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:41.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:41.458+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:41.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:41.458+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:41.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:41.458+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:41.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:41.459+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:42.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:41.820+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:42.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:41.820+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:42.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:41.820+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:42.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:41.821+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:42.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:42.416+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:42.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:42.416+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:42.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:42.416+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:42.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:42.416+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:42.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:42.447+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:42.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:42.447+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:42.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:42.447+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:42.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:42.448+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:43.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:42.804+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:43.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:42.804+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:43.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:42.804+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:43.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:42.804+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:43.781 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:43.442+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:43.782 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:43.442+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:43.782 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:43.442+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:43.782 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:43.443+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:43.782 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:43.466+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:43.782 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:43.466+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:43.782 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:43.466+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:43.783 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:43.467+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:44.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:31:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:31:43.781+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16286 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:31:44.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:43.780+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:44.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:43.780+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:44.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:43.780+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:44.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:43.782+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:44.755 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:44.485+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:44.755 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:44.485+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:44.755 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:44.485+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:44.755 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:44.485+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:44.755 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:44.443+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:44.756 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:44.443+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:44.756 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:44.443+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:44.756 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:44.444+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:45.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:44.753+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:45.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:44.753+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:45.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:44.753+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:45.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:44.755+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:45.758 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:45.453+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:45.758 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:45.453+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:45.759 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:45.453+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:45.759 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:45.454+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:45.759 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:45.472+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:45.759 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:45.472+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:45.759 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:45.472+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:45.759 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:45.472+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:46.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:45.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:46.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:45.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:46.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:45.757+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:46.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:45.758+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:46.718 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:46.473+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:46.718 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:46.473+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:46.718 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:46.473+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:46.719 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:46.474+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:46.719 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:46.450+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:46.719 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:46.450+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:46.719 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:46.450+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:46.719 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:46.450+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:46.976 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:46.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:46.976 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:46.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:46.976 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:46.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:46.976 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:46.718+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:47.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:47.405+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:47.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:47.405+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:47.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:47.405+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:47.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:47.406+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:47.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:47.440+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:47.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:47.440+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:47.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:47.440+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:47.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:47.441+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:48.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:47.719+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:48.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:47.719+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:48.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:47.719+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:48.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:47.720+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:48.746 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:48.444+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:48.746 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:48.444+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:48.747 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:48.444+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:48.747 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:48.444+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:48.747 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:48.483+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:48.747 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:48.483+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:48.747 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:48.483+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:48.747 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:48.483+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:49.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:31:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:31:48.782+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16339 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:31:49.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:48.745+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:49.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:48.745+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:49.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:48.745+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:49.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:48.746+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:49.754 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:49.446+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:49.754 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:49.446+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:49.754 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:49.446+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:49.754 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:49.447+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:49.755 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:49.499+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:49.755 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:49.499+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:49.755 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:49.499+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:49.755 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:49.499+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:50.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:49.753+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:50.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:49.753+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:50.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:49.753+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:50.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:49.753+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:50.710 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:50.485+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:50.710 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:50.485+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:50.710 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:50.485+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:50.710 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:50.485+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:50.711 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:50.448+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:50.711 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:50.448+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:50.711 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:50.448+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:50.711 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:50.449+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:51.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:50.709+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:51.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:50.709+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:51.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:50.709+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:51.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:50.710+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:51.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:51.458+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:51.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:51.458+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:51.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:51.458+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:51.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:51.459+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:51.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:51.675+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:51.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:51.675+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:51.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:51.675+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:51.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:51.677+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:51.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:51.526+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:51.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:51.526+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:51.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:51.526+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:51.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:51.526+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:52.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:52.715+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:52.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:52.715+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:52.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:52.715+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:52.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:52.716+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:52.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:52.490+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:52.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:52.490+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:52.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:52.490+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:52.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:52.490+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:52.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:52.534+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:52.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:52.534+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:52.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:52.534+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:52.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:52.535+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:53.750 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:53.519+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:53.750 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:53.519+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:53.750 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:53.519+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:53.750 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:53.520+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:53.750 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:53.449+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:53.750 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:53.449+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:53.751 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:53.449+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:53.751 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:53.449+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:54.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:31:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:31:53.784+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16391 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:31:54.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:53.764+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:54.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:53.764+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:54.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:53.764+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:54.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:53.765+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:54.738 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:54.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:54.738 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:54.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:54.738 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:54.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:54.738 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:54.477+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:54.739 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:54.477+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:54.739 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:54.477+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:54.739 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:54.478+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:54.739 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:54.534+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:54.739 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:54.534+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:54.739 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:54.534+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:54.739 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:54.535+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:55.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:54.738+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:55.755 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:55.553+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:55.755 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:55.553+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:55.755 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:55.553+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:55.755 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:55.554+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:55.756 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:55.470+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:55.756 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:55.470+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:55.756 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:55.470+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:55.756 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:55.471+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:56.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:55.754+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:56.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:55.754+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:56.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:55.754+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:56.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:55.755+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:56.753 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:56.603+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:56.753 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:56.603+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:56.753 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:56.603+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:56.753 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:56.604+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:56.754 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:56.483+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:56.754 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:56.483+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:56.754 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:56.483+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:56.754 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:56.483+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:57.055 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:56.752+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:57.056 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:56.752+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:57.056 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:56.752+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:57.056 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:56.753+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:57.774 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:57.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:57.774 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:57.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:57.774 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:57.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:57.775 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:57.561+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:57.775 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:57.434+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:57.775 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:57.434+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:57.775 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:57.434+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:57.775 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:57.435+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:58.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:57.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:58.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:57.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:58.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:57.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:58.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:57.774+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:58.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:58.514+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:58.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:58.514+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:58.514+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:58.515+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:58.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:58.411+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:58.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:58.411+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:58.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:58.411+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:58.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:58.413+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:59.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:31:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:31:58.785+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16444 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:31:59.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:58.811+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:59.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:58.811+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:59.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:58.811+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:31:59.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:58.812+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:31:59.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:59.373+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:59.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:59.373+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:59.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:59.373+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:31:59.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:31:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:31:59.374+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:31:59.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:59.515+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:59.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:59.515+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:59.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:59.515+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:31:59.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:31:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:31:59.516+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:00.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:59.840+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:00.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:59.840+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:00.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:59.840+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:00.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:31:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:31:59.840+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:00.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:00.397+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:00.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:00.397+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:00.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:00.397+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:00.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:00.397+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:00.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:00.566+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:00.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:00.566+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:00.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:00.566+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:00.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:00.567+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:01.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:00.795+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:01.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:00.795+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:01.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:00.795+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:01.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:00.795+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:01.522 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:01.359+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:01.522 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:01.359+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:01.523 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:01.359+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:01.523 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:01.360+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:01.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:01.765+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:01.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:01.765+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:01.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:01.765+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:01.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:01.765+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:01.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:01.521+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:01.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:01.521+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:01.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:01.521+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:01.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:01.522+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:02.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:02.554+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:02.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:02.554+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:02.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:02.554+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:02.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:02.555+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:02.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:02.395+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:02.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:02.395+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:02.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:02.395+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:02.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:02.396+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:03.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:02.792+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:03.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:02.792+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:03.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:02.792+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:03.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:02.793+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:03.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:03.400+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:03.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:03.400+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:03.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:03.400+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:03.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:03.401+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:03.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:03.547+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:03.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:03.547+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:03.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:03.547+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:03.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:03.548+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:04.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:32:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:32:03.787+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16496 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:32:04.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:03.793+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:04.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:03.793+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:04.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:03.793+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:04.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:03.794+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:04.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:04.372+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:04.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:04.372+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:04.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:04.372+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:04.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:04.373+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:04.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:04.566+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:04.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:04.566+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:04.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:04.566+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:04.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:04.567+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:05.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:04.772+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:05.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:04.772+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:05.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:04.772+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:05.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:04.773+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:05.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:05.363+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:05.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:05.363+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:05.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:05.363+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:05.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:05.364+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:05.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:05.528+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:05.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:05.528+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:05.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:05.528+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:05.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:05.529+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:06.038 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:05.784+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:06.038 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:05.784+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:06.038 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:05.784+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:06.038 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:05.785+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:06.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:06.399+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:06.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:06.399+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:06.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:06.399+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:06.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:06.400+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:06.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:06.520+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:06.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:06.520+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:06.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:06.520+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:06.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:06.521+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:07.134 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:06.827+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:07.134 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:06.827+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:07.134 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:06.827+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:07.134 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:06.828+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:07.817 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:07.435+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:07.818 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:07.435+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:07.818 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:07.435+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:07.818 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:07.435+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:07.818 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:07.522+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:07.818 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:07.522+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:07.819 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:07.522+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:07.819 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:07.523+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:08.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:07.816+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:08.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:07.816+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:08.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:07.816+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:08.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:07.817+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:08.790 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:08.466+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:08.790 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:08.466+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:08.790 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:08.466+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:08.790 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:08.467+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:08.791 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:08.568+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:08.791 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:08.568+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:08.791 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:08.568+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:08.791 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:08.569+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:09.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:32:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:32:08.789+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16549 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:32:09.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:08.838+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:09.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:08.838+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:09.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:08.838+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:09.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:08.839+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:09.855 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:09.469+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:09.856 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:09.469+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:09.856 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:09.469+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:09.856 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:09.470+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:09.856 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:09.528+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:09.856 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:09.528+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:09.856 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:09.528+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:09.857 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:09.529+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:10.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:09.854+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:10.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:09.854+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:10.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:09.854+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:10.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:09.855+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:10.813 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:10.443+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:10.813 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:10.443+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:10.813 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:10.443+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:10.813 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:10.444+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:10.814 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:10.504+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:10.814 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:10.504+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:10.814 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:10.504+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:10.814 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:10.505+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:11.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:10.812+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:11.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:10.812+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:11.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:10.812+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:11.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:10.813+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:11.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:11.545+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:11.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:11.545+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:11.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:11.545+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:11.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:11.546+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:11.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:11.400+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:11.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:11.400+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:11.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:11.400+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:11.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:11.401+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:12.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:11.854+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:12.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:11.854+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:12.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:11.854+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:12.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:11.855+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:12.811 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:12.498+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:12.811 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:12.498+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:12.812 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:12.498+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:12.812 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:12.499+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:12.812 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:12.441+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:12.812 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:12.441+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:12.812 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:12.441+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:12.812 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:12.442+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:13.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:12.810+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:13.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:12.810+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:13.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:12.810+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:13.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:12.811+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:13.790 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:13.457+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:13.791 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:13.457+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:13.791 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:13.457+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:13.791 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:13.458+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:13.791 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:13.504+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:13.791 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:13.504+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:13.791 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:13.504+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:13.792 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:13.504+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:14.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:32:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:32:13.791+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16601 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:32:14.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:13.789+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:14.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:13.789+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:14.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:13.789+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:14.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:13.790+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:14.751 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:14.456+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:14.752 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:14.456+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:14.752 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:14.456+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:14.752 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:14.457+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:14.752 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:14.464+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:14.752 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:14.464+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:14.753 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:14.464+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:14.753 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:14.465+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:15.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:14.750+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:15.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:14.750+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:15.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:14.750+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:15.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:14.751+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:15.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:15.484+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:15.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:15.484+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:15.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:15.484+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:15.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:15.485+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:15.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:15.430+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:15.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:15.430+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:15.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:15.430+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:15.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:15.430+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:16.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:15.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:16.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:15.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:16.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:15.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:16.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:15.719+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:16.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:16.530+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:16.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:16.530+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:16.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:16.530+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:16.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:16.531+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:16.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:16.406+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:16.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:16.406+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:16.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:16.406+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:16.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:16.407+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:16.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:16.670+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:16.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:16.670+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:16.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:16.670+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:16.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:16.671+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:17.496 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:17.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:17.497 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:17.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:17.497 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:17.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:17.497 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:17.390+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:17.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:17.495+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:17.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:17.495+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:17.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:17.495+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:17.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:17.496+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:17.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:17.689+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:17.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:17.689+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:17.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:17.689+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:17.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:17.690+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:18.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:18.462+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:18.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:18.462+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:18.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:18.462+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:18.672 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:18.463+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:18.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:18.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:18.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:18.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:18.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:18.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:18.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:18.345+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:18.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:32:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:32:18.792+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16654 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:32:18.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:18.671+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:18.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:18.671+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:18.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:18.671+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:18.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:18.671+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:19.649 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:19.451+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:19.649 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:19.451+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:19.649 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:19.451+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:19.649 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:19.453+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:19.650 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:19.296+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:19.650 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:19.296+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:19.650 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:19.296+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:19.650 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:19.297+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:19.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:19.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:19.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:19.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:19.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:19.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:19.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:19.649+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:20.652 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:20.496+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:20.652 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:20.496+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:20.652 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:20.496+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:20.652 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:20.498+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:20.652 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:20.297+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:20.653 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:20.297+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:20.653 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:20.297+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:20.653 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:20.298+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:20.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:20.650+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:20.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:20.650+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:20.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:20.650+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:20.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:20.651+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:21.570 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:21.267+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:21.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:21.267+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:21.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:21.267+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:21.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:21.268+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:21.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:21.449+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:21.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:21.449+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:21.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:21.449+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:21.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:21.450+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:21.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:21.695+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:21.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:21.695+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:21.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:21.695+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:21.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:21.696+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:22.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:22.454+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:22.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:22.454+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:22.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:22.454+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:22.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:22.455+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:22.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:22.280+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:22.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:22.280+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:22.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:22.280+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:22.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:22.280+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:23.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:22.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:23.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:22.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:23.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:22.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:23.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:22.693+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:23.631 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:23.468+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:23.631 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:23.468+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:23.632 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:23.468+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:23.632 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:23.469+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:23.632 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:23.304+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:23.632 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:23.304+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:23.632 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:23.304+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:23.633 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:23.304+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:23.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:32:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:32:23.794+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16706 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:32:23.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:23.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:23.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:23.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:23.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:23.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:23.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:23.707+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:24.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:24.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:24.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:24.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:24.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:24.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:24.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:24.295+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:24.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:24.503+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:24.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:24.503+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:24.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:24.503+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:24.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:24.504+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:25.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:24.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:25.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:24.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:25.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:24.737+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:25.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:24.738+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:25.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:25.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:25.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:25.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:25.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:25.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:25.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:25.309+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:25.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:25.533+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:25.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:25.533+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:25.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:25.533+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:25.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:25.534+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:26.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:25.735+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:26.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:25.735+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:26.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:25.735+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:26.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:25.737+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:26.574 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:26.297+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:26.575 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:26.297+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:26.575 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:26.297+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:26.575 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:26.297+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:26.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:26.573+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:26.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:26.573+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:26.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:26.573+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:26.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:26.574+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:26.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:26.690+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:26.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:26.690+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:26.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:26.690+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:26.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:26.691+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:27.620 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:27.320+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:27.620 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:27.320+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:27.621 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:27.320+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:27.621 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:27.321+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:27.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:27.619+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:27.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:27.619+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:27.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:27.619+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:27.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:27.620+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:27.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:27.738+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:27.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:27.738+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:27.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:27.738+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:27.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:27.739+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:28.666 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:28.369+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:28.666 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:28.369+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:28.666 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:28.369+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:28.666 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:28.370+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:28.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:32:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:32:28.796+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16759 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:32:28.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:28.664+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:28.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:28.664+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:28.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:28.664+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:28.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:28.665+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:28.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:28.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:28.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:28.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:28.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:28.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:28.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:28.717+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:29.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:29.335+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:29.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:29.335+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:29.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:29.335+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:29.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:29.335+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:30.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:29.763+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:30.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:29.763+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:30.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:29.763+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:30.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:29.763+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:30.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:29.694+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:30.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:29.694+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:30.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:29.694+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:30.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:29.695+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:30.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:30.327+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:30.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:30.327+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:30.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:30.327+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:30.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:30.329+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:31.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:30.738+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:31.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:30.738+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:31.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:30.738+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:31.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:30.739+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:31.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:30.749+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:31.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:30.749+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:31.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:30.749+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:31.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:30.749+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:31.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:31.352+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:31.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:31.352+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:31.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:31.352+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:31.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:31.353+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:31.923 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:31.718+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:31.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:31.718+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:31.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:31.718+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:31.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:31.719+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:31.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:31.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:31.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:31.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:31.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:31.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:31.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:31.713+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:32.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:32.388+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:32.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:32.388+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:32.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:32.388+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:32.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:32.389+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:33.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:32.755+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:33.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:32.755+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:33.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:32.755+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:33.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:32.756+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:33.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:32.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:33.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:32.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:33.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:32.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:33.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:32.722+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:33.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:33.418+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:33.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:33.418+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:33.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:33.418+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:33.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:33.419+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:34.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:32:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:32:33.797+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16811 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:32:34.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:33.800+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:34.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:33.800+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:34.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:33.800+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:34.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:33.801+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:34.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:33.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:34.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:33.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:34.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:33.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:34.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:33.705+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:34.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:34.413+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:34.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:34.413+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:34.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:34.413+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:34.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:34.414+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:35.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:34.711+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:35.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:34.711+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:35.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:34.711+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:35.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:34.712+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:35.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:34.792+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:35.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:34.792+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:35.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:34.792+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:35.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:34.793+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:35.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:35.414+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:35.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:35.414+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:35.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:35.414+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:35.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:35.415+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:36.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:35.679+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:36.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:35.679+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:36.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:35.679+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:36.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:35.680+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:36.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:35.776+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:36.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:35.776+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:36.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:35.776+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:36.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:35.777+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:36.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:36.416+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:36.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:36.416+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:36.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:36.416+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:36.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:36.417+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:37.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:36.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:37.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:36.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:37.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:36.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:37.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:36.704+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:37.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:36.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:37.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:36.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:37.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:36.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:37.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:36.806+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:37.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:37.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:37.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:37.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:37.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:37.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:37.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:37.391+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:38.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:37.675+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:38.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:37.675+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:38.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:37.675+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:38.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:37.677+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:38.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:37.828+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:38.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:37.828+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:38.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:37.828+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:38.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:37.829+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:38.658 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:38.349+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:38.658 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:38.349+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:38.659 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:38.349+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:38.659 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:38.350+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:38.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:32:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:32:38.799+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16864 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:32:38.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:38.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:38.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:38.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:38.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:38.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:38.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:38.846+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:38.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:38.657+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:38.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:38.657+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:38.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:38.657+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:38.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:38.658+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:39.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:39.368+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:39.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:39.368+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:39.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:39.368+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:39.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:39.368+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:40.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:39.852+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:40.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:39.852+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:40.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:39.852+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:40.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:39.853+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:40.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:39.682+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:40.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:39.682+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:40.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:39.682+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:40.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:39.682+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:40.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:40.327+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:40.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:40.327+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:40.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:40.327+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:40.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:40.328+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:41.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:40.901+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:41.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:40.901+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:41.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:40.901+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:41.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:40.902+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:41.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:40.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:41.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:40.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:41.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:40.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:41.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:40.717+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:41.570 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:41.282+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:41.570 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:41.282+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:41.570 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:41.282+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:41.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:41.282+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:41.899 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:41.743+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:41.900 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:41.743+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:41.900 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:41.743+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:41.900 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:41.744+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:42.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:41.898+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:42.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:41.898+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:42.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:41.898+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:42.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:41.899+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:42.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:42.249+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:42.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:42.249+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:42.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:42.249+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:42.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:42.250+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:43.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:42.859+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:43.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:42.859+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:43.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:42.859+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:43.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:42.859+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:43.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:42.705+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:43.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:42.705+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:43.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:42.705+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:43.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:42.706+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:43.632 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:43.265+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:43.633 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:43.265+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:43.633 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:43.265+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:43.633 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:43.266+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:43.899 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:32:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:32:43.801+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16916 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:32:43.899 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:43.722+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:43.900 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:43.722+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:43.900 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:43.722+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:43.900 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:43.723+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:44.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:43.898+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:44.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:43.898+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:44.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:43.898+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:44.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:43.899+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:44.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:44.222+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:44.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:44.222+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:44.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:44.222+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:44.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:44.223+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:45.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:44.887+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:45.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:44.887+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:45.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:44.887+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:45.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:44.889+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:45.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:44.753+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:45.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:44.753+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:45.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:44.753+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:45.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:44.753+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:45.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:45.201+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:45.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:45.201+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:45.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:45.201+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:45.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:45.202+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:46.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:45.876+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:46.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:45.876+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:46.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:45.876+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:46.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:45.876+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:46.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:45.770+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:46.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:45.770+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:46.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:45.770+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:46.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:45.771+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:46.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:46.235+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:46.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:46.235+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:46.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:46.235+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:46.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:46.236+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:47.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:46.818+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:47.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:46.818+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:47.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:46.818+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:47.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:46.819+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:47.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:46.858+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:47.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:46.858+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:47.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:46.858+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:47.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:46.859+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:47.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:47.284+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:47.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:47.284+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:47.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:47.284+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:47.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:47.285+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:48.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:47.885+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:48.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:47.886+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:48.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:47.886+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:48.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:47.887+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:48.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:47.869+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:48.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:47.869+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:48.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:47.869+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:48.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:47.869+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:48.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:48.236+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:48.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:48.236+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:48.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:48.236+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:48.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:48.237+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:49.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:32:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:32:48.803+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 16969 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:32:49.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:48.872+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:49.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:48.872+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:49.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:48.872+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:49.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:48.873+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:49.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:48.887+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:49.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:48.887+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:49.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:48.887+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:49.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:48.888+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:49.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:49.225+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:49.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:49.225+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:49.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:49.225+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:49.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:49.226+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:50.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:49.907+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:50.177 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:49.907+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:50.178 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:49.907+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:50.178 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:49.908+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:50.178 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:49.908+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:50.178 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:49.908+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:50.178 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:49.908+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:50.178 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:49.909+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:50.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:50.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:50.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:50.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:50.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:50.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:50.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:50.176+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:51.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:50.907+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:51.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:50.907+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:51.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:50.907+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:51.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:50.908+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:51.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:50.905+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:51.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:50.905+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:51.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:50.905+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:51.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:50.906+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:51.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:51.179+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:51.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:51.179+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:51.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:51.179+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:51.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:51.179+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:52.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:51.956+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:52.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:51.956+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:52.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:51.956+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:52.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:51.957+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:52.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:51.888+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:52.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:51.888+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:52.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:51.888+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:52.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:51.889+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:52.487 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:52.182+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:52.487 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:52.182+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:52.487 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:52.182+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:52.487 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:52.183+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:53.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:52.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:53.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:52.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:53.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:52.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:53.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:52.995+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:53.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:52.905+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:53.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:52.905+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:53.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:52.905+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:53.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:52.906+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:53.634 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:53.188+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:53.634 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:53.188+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:53.634 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:53.188+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:53.634 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:53.189+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:53.891 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:32:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:32:53.804+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17021 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:32:54.163 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:53.889+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:54.163 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:53.889+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:54.164 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:53.889+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:54.164 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:53.889+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:54.164 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:53.972+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:54.164 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:53.972+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:54.164 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:53.972+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:54.164 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:53.972+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:54.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:54.161+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:54.423 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:54.161+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:54.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:54.161+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:54.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:54.161+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:55.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:54.908+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:55.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:54.908+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:55.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:54.908+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:55.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:54.909+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:55.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:54.999+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:55.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:54.999+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:55.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:54.999+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:55.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:55.000+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:55.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:55.199+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:55.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:55.199+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:55.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:55.199+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:55.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:55.199+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:56.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:55.910+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:56.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:55.910+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:56.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:55.910+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:56.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:55.911+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:56.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:55.954+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:56.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:55.954+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:56.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:55.954+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:56.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:55.954+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:56.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:56.238+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:56.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:56.238+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:56.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:56.238+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:56.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:56.239+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:57.234 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:56.995+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:57.235 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:56.995+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:57.235 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:56.995+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:57.235 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:56.996+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:57.235 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:56.957+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:57.235 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:56.957+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:57.235 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:56.957+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:57.236 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:56.958+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:57.523 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:57.232+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:57.523 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:57.232+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:57.523 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:57.232+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:57.523 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:57.233+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:58.256 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:57.926+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:58.256 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:57.926+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:58.256 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:57.926+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:58.256 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:57.927+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:58.257 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:57.995+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:58.257 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:57.995+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:58.257 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:57.995+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:58.257 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:57.995+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:58.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:58.254+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:58.254+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:58.254+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:58.254+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:59.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:32:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:32:58.805+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17074 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:32:59.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:58.971+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:58.971+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:58.971+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:32:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:58.971+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:32:59.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:58.917+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:59.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:58.917+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:59.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:58.917+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:32:59.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:58.918+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:32:59.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:59.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:59.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:59.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:59.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:59.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:32:59.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:32:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:32:59.294+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:00.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:59.904+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:00.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:59.904+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:00.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:59.904+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:00.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:32:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:32:59.904+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:00.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:59.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:00.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:59.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:00.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:59.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:00.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:32:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:32:59.975+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:00.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:00.257+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:00.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:00.257+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:00.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:00.257+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:00.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:00.257+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:01.287 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:01.007+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:01.288 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:01.007+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:01.288 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:01.007+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:01.288 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:01.008+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:01.288 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:00.923+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:01.288 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:00.923+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:01.288 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:00.923+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:01.289 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:00.924+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:01.570 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:01.285+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:01.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:01.285+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:01.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:01.285+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:01.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:01.286+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:02.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:01.961+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:02.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:01.961+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:02.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:01.961+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:02.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:01.961+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:02.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:01.897+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:02.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:01.897+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:02.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:01.897+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:02.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:01.897+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:02.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:02.322+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:02.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:02.322+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:02.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:02.322+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:02.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:02.322+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:03.351 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:02.985+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:03.352 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:02.985+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:03.352 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:02.985+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:03.352 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:02.985+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:03.352 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:02.935+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:03.352 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:02.935+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:03.352 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:02.935+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:03.353 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:02.936+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:03.634 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:03.349+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:03.634 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:03.349+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:03.634 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:03.349+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:03.635 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:03.349+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:03.913 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:33:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:33:03.808+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17126 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:33:04.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:03.912+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:04.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:03.912+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:04.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:03.912+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:04.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:03.913+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:04.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:03.981+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:04.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:03.981+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:04.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:03.981+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:04.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:03.982+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:04.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:04.331+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:04.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:04.331+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:04.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:04.331+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:04.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:04.332+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:05.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:04.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:05.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:04.974+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:05.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:04.974+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:05.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:04.975+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:05.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:04.879+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:05.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:04.879+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:05.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:04.879+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:05.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:04.880+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:05.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:05.320+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:05.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:05.320+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:05.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:05.320+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:05.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:05.321+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:06.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:05.936+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:06.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:05.936+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:06.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:05.936+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:06.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:05.937+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:06.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:05.905+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:06.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:05.905+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:06.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:05.905+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:06.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:05.905+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:06.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:06.298+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:06.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:06.298+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:06.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:06.298+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:06.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:06.299+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:07.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:06.884+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:07.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:06.884+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:07.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:06.884+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:07.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:06.884+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:07.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:06.984+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:07.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:06.984+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:07.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:06.984+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:07.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:06.985+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:07.596 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:07.293+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:07.597 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:07.293+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:07.597 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:07.293+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:07.597 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:07.294+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:08.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:07.964+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:08.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:07.964+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:08.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:07.964+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:08.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:07.965+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:08.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:07.853+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:08.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:07.853+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:08.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:07.853+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:08.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:07.854+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:08.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:08.314+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:08.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:08.315+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:08.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:08.315+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:08.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:08.315+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:09.173 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:33:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:33:08.810+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17179 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:33:09.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:08.853+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:09.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:08.853+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:09.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:08.853+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:09.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:08.854+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:09.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:08.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:09.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:08.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:09.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:08.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:09.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:08.979+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:09.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:09.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:09.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:09.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:09.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:09.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:09.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:09.345+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:10.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:09.894+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:10.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:09.894+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:10.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:09.894+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:10.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:09.895+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:10.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:10.002+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:10.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:10.002+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:10.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:10.002+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:10.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:10.002+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:10.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:10.318+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:10.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:10.318+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:10.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:10.318+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:10.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:10.318+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:11.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:10.872+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:11.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:10.872+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:11.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:10.872+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:11.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:10.873+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:11.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:10.971+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:11.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:10.971+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:11.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:10.971+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:11.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:10.971+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:11.570 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:11.312+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:11.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:11.312+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:11.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:11.312+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:11.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:11.313+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:12.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:11.954+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:12.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:11.954+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:12.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:11.954+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:12.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:11.954+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:12.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:11.909+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:12.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:11.909+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:12.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:11.909+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:12.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:11.910+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:12.633 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:12.360+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:12.633 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:12.360+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:12.633 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:12.360+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:12.633 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:12.360+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:13.335 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:12.957+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:13.335 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:12.957+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:13.335 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:12.957+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:13.335 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:12.958+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:13.336 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:12.952+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:13.336 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:12.952+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:13.336 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:12.952+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:13.336 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:12.952+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:13.635 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:13.333+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:13.635 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:13.333+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:13.635 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:13.333+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:13.635 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:13.334+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:13.918 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:33:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:33:13.812+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17231 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:33:14.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:13.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:14.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:13.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:14.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:13.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:14.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:13.974+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:14.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:13.917+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:14.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:13.917+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:14.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:13.917+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:14.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:13.918+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:14.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:14.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:14.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:14.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:14.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:14.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:14.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:14.309+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:15.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:14.999+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:15.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:14.999+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:15.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:14.999+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:15.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:15.000+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:15.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:14.901+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:15.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:14.901+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:15.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:14.901+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:15.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:14.902+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:15.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:15.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:15.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:15.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:15.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:15.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:15.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:15.341+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:16.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:15.873+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:16.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:15.873+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:16.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:15.873+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:16.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:15.874+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:16.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:15.965+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:16.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:15.965+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:16.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:15.965+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:16.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:15.966+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:16.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:16.376+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:16.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:16.376+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:16.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:16.376+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:16.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:16.378+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:16.977 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:16.874+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:16.977 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:16.874+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:16.977 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:16.874+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:16.977 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:16.875+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:17.391 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:16.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:17.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:16.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:17.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:16.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:17.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:16.976+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:17.668 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:17.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:17.669 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:17.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:17.669 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:17.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:17.669 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:17.391+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:17.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:17.828+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:17.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:17.828+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:17.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:17.828+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:17.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:17.829+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:18.345 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:18.011+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:18.345 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:18.011+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:18.345 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:18.011+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:18.345 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:18.012+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:18.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:18.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:18.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:18.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:18.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:18.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:18.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:18.344+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:19.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:33:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:33:18.813+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17284 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:33:19.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:19.057+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:19.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:19.057+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:19.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:19.057+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:19.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:19.058+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:19.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:18.817+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:19.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:18.817+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:19.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:18.817+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:19.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:18.818+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:19.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:19.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:19.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:19.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:19.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:19.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:19.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:19.391+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:20.071 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:19.774+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:20.071 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:19.774+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:20.071 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:19.774+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:20.071 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:19.774+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:20.398 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:20.069+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:20.399 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:20.069+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:20.399 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:20.069+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:20.399 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:20.071+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:20.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:20.397+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:20.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:20.397+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:20.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:20.397+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:20.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:20.398+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:21.093 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:20.781+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:21.093 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:20.781+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:21.093 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:20.781+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:21.094 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:20.782+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:21.418 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:21.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:21.418 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:21.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:21.418 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:21.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:21.418 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:21.093+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:21.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:21.416+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:21.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:21.416+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:21.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:21.416+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:21.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:21.417+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:22.095 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:21.810+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:22.095 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:21.810+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:22.096 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:21.810+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:22.096 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:21.810+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:22.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:22.094+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:22.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:22.094+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:22.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:22.094+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:22.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:22.095+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:22.710 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:22.437+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:22.710 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:22.437+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:22.710 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:22.437+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:22.711 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:22.438+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:23.077 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:22.833+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:23.077 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:22.833+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:23.077 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:22.833+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:23.078 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:22.834+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:23.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:23.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:23.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:23.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:23.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:23.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:23.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:23.077+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:23.816 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:23.451+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:23.817 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:23.451+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:23.817 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:23.451+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:23.817 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:23.452+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:24.102 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:33:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:33:23.815+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17336 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:33:24.103 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:23.822+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:24.103 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:23.822+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:24.103 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:23.822+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:24.104 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:23.822+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:24.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:24.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:24.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:24.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:24.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:24.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:24.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:24.102+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:24.780 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:24.465+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:24.780 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:24.465+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:24.781 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:24.465+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:24.781 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:24.466+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:25.128 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:24.779+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:25.128 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:24.779+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:25.128 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:24.779+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:25.128 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:24.780+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:25.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:25.126+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:25.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:25.126+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:25.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:25.126+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:25.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:25.128+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:25.816 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:25.429+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:25.817 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:25.429+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:25.817 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:25.429+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:25.817 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:25.429+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:26.167 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:25.815+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:26.167 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:25.815+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:26.168 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:25.815+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:26.168 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:25.816+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:26.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:26.166+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:26.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:26.166+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:26.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:26.166+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:26.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:26.167+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:26.770 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:26.446+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:26.770 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:26.446+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:26.770 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:26.446+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:26.770 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:26.447+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:27.125 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:26.768+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:27.125 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:26.768+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:27.125 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:26.768+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:27.125 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:26.769+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:27.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:27.123+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:27.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:27.123+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:27.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:27.123+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:27.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:27.125+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:27.745 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:27.742+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:27.745 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:27.742+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:27.745 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:27.742+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:27.746 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:27.744+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:27.746 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:27.494+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:27.746 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:27.494+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:27.746 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:27.494+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:27.746 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:27.494+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:28.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:28.078+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:28.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:28.078+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:28.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:28.078+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:28.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:28.080+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:28.819 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:28.508+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:28.819 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:28.508+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:28.819 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:28.508+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:28.819 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:28.509+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:28.820 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:28.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:28.820 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:28.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:28.820 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:28.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:28.820 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:28.735+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:29.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:33:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:33:28.817+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17389 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:33:29.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:29.061+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:29.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:29.061+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:29.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:29.061+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:29.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:29.062+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:29.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:29.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:29.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:29.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:29.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:29.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:29.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:29.707+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:29.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:29.550+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:29.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:29.550+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:29.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:29.550+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:29.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:29.551+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:30.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:30.098+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:30.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:30.098+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:30.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:30.098+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:30.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:30.098+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:30.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:30.735+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:30.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:30.735+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:30.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:30.735+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:30.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:30.736+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:30.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:30.505+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:30.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:30.505+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:30.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:30.505+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:30.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:30.505+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:31.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:31.134+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:31.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:31.134+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:31.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:31.134+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:31.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:31.135+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:31.782 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:31.471+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:31.783 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:31.471+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:31.783 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:31.471+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:31.783 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:31.472+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:32.117 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:31.781+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:32.117 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:31.781+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:32.117 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:31.781+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:32.117 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:31.782+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:32.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:32.115+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:32.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:32.115+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:32.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:32.115+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:32.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:32.116+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:32.769 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:32.431+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:32.769 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:32.431+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:32.769 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:32.431+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:32.769 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:32.432+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:33.083 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:32.767+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:33.083 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:32.767+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:33.083 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:32.767+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:33.083 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:32.768+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:33.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:33.081+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:33.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:33.081+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:33.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:33.081+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:33.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:33.082+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:33.732 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:33.476+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:33.733 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:33.476+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:33.733 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:33.476+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:33.733 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:33.476+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:34.092 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:33:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:33:33.819+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17441 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:33:34.092 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:33.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:34.092 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:33.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:34.093 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:33.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:34.093 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:33.732+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:34.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:34.090+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:34.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:34.090+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:34.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:34.090+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:34.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:34.092+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:34.734 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:34.463+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:34.734 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:34.463+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:34.734 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:34.463+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:34.734 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:34.464+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:35.123 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:34.732+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:35.123 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:34.732+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:35.123 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:34.732+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:35.124 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:34.733+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:35.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:35.122+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:35.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:35.122+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:35.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:35.122+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:35.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:35.123+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:35.673 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:35.418+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:35.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:35.418+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:35.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:35.418+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:35.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:35.419+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:36.164 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:35.765+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:36.164 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:35.765+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:36.164 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:35.765+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:36.164 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:35.765+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:36.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:36.162+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:36.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:36.162+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:36.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:36.162+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:36.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:36.163+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:36.751 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:36.426+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:36.751 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:36.426+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:36.751 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:36.426+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:36.751 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:36.426+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:37.161 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:36.749+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:37.161 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:36.749+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:37.161 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:36.749+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:37.161 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:36.750+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:37.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:37.159+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:37.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:37.159+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:37.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:37.159+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:37.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:37.160+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:37.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:37.395+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:37.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:37.395+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:37.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:37.395+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:37.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:37.395+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:38.128 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:37.702+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:38.129 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:37.702+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:38.129 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:37.702+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:38.129 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:37.703+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:38.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:38.127+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:38.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:38.127+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:38.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:38.127+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:38.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:38.128+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:38.721 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:38.440+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:38.721 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:38.440+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:38.721 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:38.440+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:38.722 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:38.442+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:39.172 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:33:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:33:38.820+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17494 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:33:39.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:38.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:39.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:38.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:39.172 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:38.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:39.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:38.720+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:39.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:39.170+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:39.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:39.170+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:39.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:39.170+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:39.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:39.171+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:39.701 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:39.434+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:39.702 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:39.434+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:39.702 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:39.434+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:39.702 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:39.435+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:40.130 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:39.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:40.131 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:39.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:40.131 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:39.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:40.131 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:39.701+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:40.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:40.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:40.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:40.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:40.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:40.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:40.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:40.130+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:40.740 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:40.469+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:40.740 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:40.469+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:40.741 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:40.469+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:40.741 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:40.470+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:41.137 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:40.739+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:41.137 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:40.739+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:41.137 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:40.739+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:41.137 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:40.740+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:41.423 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:41.135+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:41.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:41.135+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:41.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:41.135+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:41.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:41.136+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:41.710 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:41.441+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:41.711 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:41.441+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:41.711 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:41.441+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:41.711 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:41.442+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:42.097 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:41.709+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:42.097 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:41.709+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:42.098 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:41.709+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:42.098 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:41.710+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:42.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:42.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:42.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:42.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:42.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:42.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:42.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:42.097+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:42.876 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:42.711+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:42.876 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:42.711+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:42.876 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:42.711+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:42.877 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:42.712+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:42.877 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:42.463+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:42.877 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:42.463+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:42.877 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:42.463+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:42.877 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:42.464+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:43.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:43.121+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:43.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:43.121+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:43.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:43.121+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:43.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:43.122+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:43.824 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:43.512+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:43.824 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:43.512+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:43.824 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:43.512+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:43.825 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:43.513+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:43.825 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:43.723+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:43.825 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:43.723+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:43.825 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:43.723+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:43.825 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:43.724+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:44.158 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:33:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:33:43.822+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17546 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:33:44.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:44.157+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:44.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:44.157+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:44.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:44.157+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:44.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:44.158+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:44.759 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:44.490+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:44.760 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:44.490+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:44.760 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:44.490+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:44.760 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:44.490+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:45.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:44.758+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:45.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:44.758+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:45.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:44.758+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:45.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:44.758+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:45.495 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:45.187+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:45.495 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:45.187+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:45.496 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:45.187+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:45.496 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:45.187+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:45.799 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:45.494+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:45.799 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:45.494+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:45.799 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:45.494+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:45.799 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:45.494+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:46.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:45.798+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:46.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:45.798+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:46.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:45.798+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:46.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:45.798+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:46.492 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:46.178+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:46.492 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:46.178+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:46.493 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:46.178+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:46.493 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:46.178+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:46.774 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:46.491+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:46.774 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:46.491+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:46.775 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:46.491+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:46.775 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:46.492+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:47.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:46.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:47.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:46.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:47.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:46.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:47.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:46.773+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:47.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:47.469+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:47.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:47.469+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:47.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:47.469+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:47.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:47.470+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:47.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:47.223+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:47.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:47.223+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:47.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:47.223+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:47.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:47.224+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:48.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:47.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:48.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:47.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:48.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:47.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:48.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:47.736+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:48.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:48.431+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:48.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:48.431+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:48.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:48.431+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:48.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:48.432+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:48.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:48.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:48.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:48.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:48.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:48.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:48.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:48.259+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:49.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:33:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:33:48.824+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17599 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:33:49.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:48.727+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:49.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:48.727+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:49.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:48.727+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:49.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:48.728+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:49.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:49.290+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:49.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:49.290+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:49.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:49.291+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:49.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:49.291+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:49.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:49.398+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:49.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:49.398+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:49.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:49.398+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:49.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:49.399+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:50.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:49.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:50.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:49.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:50.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:49.773+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:50.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:49.774+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:50.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:50.271+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:50.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:50.271+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:50.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:50.271+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:50.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:50.272+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:50.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:50.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:50.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:50.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:50.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:50.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:50.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:50.408+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:51.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:50.762+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:51.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:50.762+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:51.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:50.762+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:51.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:50.762+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:51.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:51.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:51.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:51.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:51.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:51.305+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:51.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:51.305+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:51.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:51.367+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:51.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:51.367+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:51.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:51.367+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:51.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:51.368+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:51.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:51.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:51.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:51.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:51.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:51.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:51.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:51.722+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:52.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:52.332+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:52.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:52.332+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:52.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:52.332+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:52.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:52.333+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:52.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:52.396+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:52.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:52.397+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:52.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:52.397+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:52.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:52.398+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:53.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:52.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:53.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:52.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:53.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:52.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:53.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:52.718+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:53.637 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:53.311+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:53.637 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:53.311+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:53.637 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:53.311+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:53.638 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:53.311+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:53.638 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:53.378+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:53.638 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:53.378+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:53.638 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:53.378+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:53.638 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:53.380+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:53.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:33:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:33:53.826+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17651 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:33:53.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:53.750+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:53.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:53.750+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:53.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:53.750+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:53.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:53.751+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:54.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:54.359+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:54.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:54.359+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:54.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:54.359+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:54.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:54.360+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:54.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:54.283+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:54.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:54.283+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:54.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:54.283+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:54.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:54.284+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:55.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:54.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:55.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:54.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:55.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:54.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:55.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:54.717+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:55.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:55.320+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:55.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:55.320+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:55.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:55.320+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:55.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:55.321+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:55.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:55.247+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:55.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:55.247+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:55.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:55.247+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:55.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:55.248+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:56.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:55.682+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:56.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:55.682+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:56.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:55.682+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:56.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:55.684+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:56.630 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:56.265+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:56.631 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:56.265+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:56.631 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:56.265+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:56.631 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:56.266+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:56.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:56.318+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:56.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:56.318+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:56.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:56.318+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:56.632 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:56.318+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:56.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:56.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:56.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:56.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:56.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:56.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:56.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:56.708+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:57.669 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:57.224+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:57.669 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:57.224+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:57.669 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:57.224+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:57.669 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:57.224+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:57.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:57.291+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:57.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:57.291+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:57.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:57.291+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:57.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:57.292+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:57.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:57.667+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:57.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:57.667+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:57.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:57.667+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:57.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:57.668+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:58.324 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:58.244+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:58.324 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:58.244+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:58.324 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:58.244+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:58.324 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:58.244+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:58.322+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:58.322+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:58.322+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:58.323+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:59.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:33:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:33:58.828+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17704 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:33:59.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:58.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:59.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:58.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:59.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:58.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:33:59.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:58.719+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:33:59.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:59.273+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:59.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:59.273+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:59.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:59.273+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:33:59.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:33:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:33:59.274+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:33:59.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:59.222+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:59.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:59.222+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:59.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:59.222+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:33:59.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:33:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:33:59.223+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:00.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:59.688+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:00.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:59.688+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:00.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:59.688+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:00.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:33:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:33:59.689+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:00.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:00.304+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:00.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:00.304+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:00.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:00.304+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:00.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:00.305+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:00.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:00.232+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:00.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:00.232+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:00.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:00.232+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:00.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:00.233+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:01.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:00.733+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:01.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:00.733+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:01.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:00.733+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:01.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:00.734+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:01.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:01.237+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:01.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:01.237+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:01.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:01.237+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:01.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:01.237+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:01.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:01.300+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:01.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:01.300+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:01.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:01.300+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:01.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:01.301+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:01.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:01.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:01.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:01.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:01.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:01.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:01.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:01.747+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:02.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:02.229+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:02.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:02.229+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:02.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:02.229+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:02.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:02.230+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:02.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:02.330+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:02.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:02.330+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:02.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:02.330+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:02.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:02.331+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:03.045 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:02.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:03.045 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:02.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:03.045 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:02.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:03.045 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:02.717+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:03.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:03.293+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:03.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:03.293+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:03.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:03.293+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:03.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:03.294+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:03.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:03.254+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:03.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:03.254+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:03.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:03.254+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:03.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:03.255+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:03.923 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:34:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:34:03.829+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17756 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:34:03.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:03.695+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:03.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:03.695+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:03.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:03.695+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:03.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:03.696+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:04.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:04.252+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:04.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:04.252+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:04.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:04.252+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:04.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:04.254+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:04.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:04.298+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:04.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:04.298+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:04.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:04.298+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:04.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:04.298+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:04.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:04.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:04.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:04.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:04.672 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:04.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:04.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:04.670+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:05.655 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:05.334+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:05.656 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:05.334+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:05.656 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:05.334+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:05.656 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:05.335+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:05.656 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:05.266+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:05.657 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:05.266+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:05.657 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:05.266+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:05.657 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:05.267+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:05.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:05.654+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:05.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:05.654+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:05.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:05.654+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:05.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:05.655+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:06.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:06.218+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:06.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:06.218+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:06.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:06.218+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:06.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:06.219+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:06.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:06.299+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:06.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:06.299+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:06.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:06.299+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:06.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:06.300+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:07.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:06.673+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:07.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:06.673+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:07.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:06.673+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:07.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:06.674+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:07.650 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:07.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:07.651 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:07.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:07.651 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:07.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:07.651 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:07.275+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:07.651 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:07.184+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:07.651 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:07.184+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:07.652 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:07.184+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:07.652 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:07.185+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:07.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:07.649+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:07.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:07.649+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:07.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:07.649+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:07.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:07.650+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:08.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:08.163+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:08.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:08.163+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:08.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:08.163+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:08.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:08.164+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:08.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:08.257+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:08.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:08.257+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:08.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:08.257+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:08.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:08.257+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:09.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:34:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:34:08.831+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17809 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:34:09.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:08.685+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:09.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:08.685+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:09.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:08.685+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:09.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:08.686+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:09.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:09.185+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:09.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:09.185+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:09.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:09.185+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:09.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:09.186+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:09.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:09.284+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:09.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:09.284+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:09.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:09.284+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:09.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:09.285+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:10.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:09.689+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:10.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:09.689+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:10.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:09.689+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:10.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:09.690+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:10.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:10.314+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:10.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:10.314+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:10.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:10.314+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:10.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:10.314+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:10.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:10.201+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:10.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:10.201+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:10.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:10.201+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:10.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:10.202+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:11.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:10.708+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:11.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:10.708+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:11.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:10.708+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:11.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:10.709+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:11.570 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:11.251+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:11.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:11.251+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:11.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:11.251+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:11.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:11.251+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:11.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:11.304+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:11.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:11.304+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:11.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:11.304+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:11.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:11.305+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:11.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:11.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:11.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:11.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:11.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:11.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:11.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:11.697+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:12.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:12.336+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:12.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:12.336+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:12.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:12.336+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:12.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:12.336+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:12.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:12.287+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:12.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:12.287+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:12.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:12.287+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:12.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:12.289+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:13.118 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:12.705+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:13.118 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:12.705+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:13.118 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:12.705+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:13.119 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:12.706+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:13.368 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:13.366+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:13.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:13.297+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:13.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:13.297+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:13.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:13.297+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:13.369 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:13.299+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:13.638 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:13.366+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:13.638 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:13.366+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:13.638 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:13.367+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:13.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:34:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:34:13.833+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17861 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:34:13.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:13.681+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:13.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:13.681+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:13.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:13.681+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:13.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:13.682+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:14.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:14.321+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:14.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:14.321+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:14.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:14.321+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:14.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:14.322+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:14.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:14.348+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:14.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:14.348+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:14.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:14.348+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:14.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:14.348+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:15.003 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:14.673+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:15.003 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:14.673+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:15.003 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:14.673+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:15.003 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:14.673+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:15.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:15.339+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:15.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:15.339+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:15.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:15.339+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:15.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:15.340+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:15.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:15.339+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:15.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:15.339+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:15.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:15.339+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:15.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:15.341+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:15.971 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:15.696+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:15.972 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:15.696+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:15.972 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:15.696+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:15.972 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:15.697+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:16.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:16.380+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:16.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:16.380+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:16.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:16.380+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:16.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:16.381+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:16.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:16.303+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:16.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:16.303+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:16.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:16.303+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:16.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:16.304+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:17.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:16.735+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:17.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:16.735+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:17.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:16.735+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:17.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:16.736+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:17.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:17.398+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:17.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:17.398+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:17.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:17.398+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:17.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:17.399+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:17.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:17.302+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:17.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:17.302+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:17.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:17.302+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:17.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:17.303+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:18.153 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:17.714+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:18.154 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:17.714+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:18.154 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:17.714+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:18.154 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:17.714+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:18.414 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:18.289+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:18.414 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:18.289+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:18.414 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:18.289+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:18.414 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:18.290+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:18.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:18.412+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:18.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:18.412+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:18.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:18.412+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:18.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:18.413+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:19.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:34:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:34:18.835+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17914 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:34:19.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:18.701+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:19.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:18.701+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:19.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:18.701+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:19.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:18.702+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:19.659 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:19.389+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:19.659 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:19.389+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:19.659 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:19.389+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:19.660 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:19.390+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:19.660 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:19.250+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:19.660 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:19.250+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:19.660 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:19.250+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:19.660 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:19.251+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:19.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:19.657+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:19.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:19.657+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:19.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:19.657+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:19.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:19.659+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:20.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:20.345+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:20.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:20.345+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:20.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:20.345+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:20.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:20.347+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:20.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:20.222+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:20.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:20.222+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:20.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:20.222+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:20.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:20.223+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:21.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:20.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:21.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:20.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:21.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:20.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:21.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:20.687+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:21.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:21.180+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:21.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:21.180+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:21.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:21.180+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:21.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:21.181+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:21.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:21.324+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:21.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:21.324+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:21.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:21.324+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:21.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:21.325+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:21.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:21.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:21.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:21.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:21.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:21.720+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:21.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:21.721+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:22.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:22.299+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:22.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:22.299+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:22.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:22.299+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:22.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:22.300+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:22.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:22.220+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:22.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:22.220+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:22.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:22.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:22.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:22.221+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:23.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:22.770+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:23.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:22.770+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:23.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:22.770+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:23.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:22.771+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:23.638 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:23.301+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:23.639 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:23.301+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:23.639 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:23.301+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:23.639 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:23.301+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:23.639 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:23.229+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:23.639 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:23.229+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:23.640 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:23.229+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:23.640 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:23.229+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:23.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:34:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:34:23.836+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 17966 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:34:23.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:23.723+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:23.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:23.723+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:23.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:23.723+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:23.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:23.724+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:24.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:24.343+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:24.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:24.343+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:24.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:24.343+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:24.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:24.343+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:24.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:24.210+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:24.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:24.210+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:24.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:24.210+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:24.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:24.210+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:25.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:24.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:25.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:24.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:25.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:24.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:25.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:24.679+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:25.488 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:25.310+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:25.489 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:25.310+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:25.489 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:25.310+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:25.489 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:25.311+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:25.490 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:25.189+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:25.490 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:25.189+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:25.490 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:25.189+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:25.490 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:25.190+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:25.818 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:25.647+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:25.818 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:25.647+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:25.819 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:25.647+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:25.819 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:25.647+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:26.395 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:26.141+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:26.396 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:26.141+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:26.396 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:26.141+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:26.396 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:26.141+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:26.396 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:26.271+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:26.396 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:26.271+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:26.396 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:26.271+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:26.397 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:26.273+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:26.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:26.621+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:26.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:26.621+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:26.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:26.621+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:26.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:26.622+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:27.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:27.256+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:27.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:27.256+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:27.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:27.256+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:27.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:27.257+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:27.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:27.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:27.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:27.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:27.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:27.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:27.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:27.163+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:27.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:27.606+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:27.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:27.606+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:27.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:27.606+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:27.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:27.607+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:28.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:28.217+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:28.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:28.217+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:28.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:28.217+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:28.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:28.219+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:28.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:28.151+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:28.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:28.151+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:28.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:28.151+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:28.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:28.152+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:28.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:34:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:34:28.838+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18019 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:34:28.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:28.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:28.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:28.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:28.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:28.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:28.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:28.627+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:29.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:29.176+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:29.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:29.176+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:29.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:29.176+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:29.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:29.176+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:29.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:29.160+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:29.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:29.160+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:29.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:29.160+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:29.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:29.161+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:30.116 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:29.673+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:30.116 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:29.673+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:30.117 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:29.673+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:30.117 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:29.674+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:30.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:30.226+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:30.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:30.226+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:30.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:30.226+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:30.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:30.227+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:30.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:30.114+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:30.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:30.114+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:30.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:30.114+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:30.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:30.114+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:31.151 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:30.674+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:31.151 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:30.674+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:31.151 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:30.674+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:31.151 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:30.675+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:31.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:31.205+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:31.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:31.205+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:31.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:31.205+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:31.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:31.207+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:31.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:31.148+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:31.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:31.148+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:31.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:31.148+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:31.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:31.149+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:31.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:31.645+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:31.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:31.645+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:31.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:31.645+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:31.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:31.646+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:32.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:32.176+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:32.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:32.176+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:32.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:32.176+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:32.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:32.177+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:32.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:32.167+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:32.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:32.167+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:32.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:32.167+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:32.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:32.168+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:32.643+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:32.643+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:32.643+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:32.644+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:33.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:33.196+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:33.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:33.196+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:33.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:33.196+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:33.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:33.197+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:33.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:33.158+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:33.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:33.158+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:33.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:33.158+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:33.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:33.158+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:33.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:34:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:34:33.839+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18071 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:34:33.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:33.640+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:33.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:33.640+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:33.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:33.640+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:33.641+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:34.601 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:34.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:34.601 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:34.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:34.601 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:34.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:34.601 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:34.215+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:34.602 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:34.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:34.602 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:34.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:34.602 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:34.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:34.602 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:34.204+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:34.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:34.598+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:34.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:34.598+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:34.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:34.598+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:34.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:34.599+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:35.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:35.252+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:35.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:35.252+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:35.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:35.252+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:35.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:35.252+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:35.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:35.156+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:35.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:35.156+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:35.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:35.156+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:35.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:35.156+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:35.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:35.607+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:35.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:35.607+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:35.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:35.607+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:35.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:35.608+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:36.567 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:36.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:36.567 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:36.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:36.567 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:36.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:36.567 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:36.176+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:36.567 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:36.209+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:36.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:36.209+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:36.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:36.209+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:36.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:36.209+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:36.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:36.607+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:36.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:36.607+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:36.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:36.607+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:36.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:36.608+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:37.284 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:37.165+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:37.284 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:37.165+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:37.284 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:37.165+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:37.284 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:37.166+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:37.284 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:37.130+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:37.285 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:37.130+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:37.285 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:37.130+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:37.285 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:37.130+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:37.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:37.566+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:37.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:37.566+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:37.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:37.566+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:37.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:37.566+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:38.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:38.189+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:38.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:38.189+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:38.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:38.189+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:38.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:38.190+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:38.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:38.139+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:38.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:38.139+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:38.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:38.139+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:38.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:38.140+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:38.844 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:38.539+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:38.844 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:38.539+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:38.844 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:38.539+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:38.845 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:38.540+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:39.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:34:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:34:38.842+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18124 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:34:39.590 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:39.182+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:39.590 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:39.182+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:39.590 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:39.182+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:39.590 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:39.182+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:39.590 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:39.181+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:39.591 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:39.181+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:39.591 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:39.181+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:39.591 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:39.182+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:39.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:39.588+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:39.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:39.588+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:39.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:39.588+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:39.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:39.589+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:40.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:40.156+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:40.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:40.157+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:40.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:40.157+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:40.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:40.158+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:40.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:40.160+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:40.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:40.160+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:40.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:40.160+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:40.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:40.161+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:40.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:40.565+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:40.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:40.565+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:40.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:40.565+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:40.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:40.565+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:41.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:41.168+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:41.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:41.169+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:41.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:41.169+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:41.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:41.169+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:41.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:41.164+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:41.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:41.164+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:41.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:41.164+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:41.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:41.166+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:41.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:41.518+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:41.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:41.518+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:41.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:41.518+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:41.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:41.518+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:42.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:42.173+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:42.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:42.173+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:42.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:42.173+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:42.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:42.174+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:42.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:42.135+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:42.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:42.135+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:42.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:42.135+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:42.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:42.136+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:42.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:42.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:42.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:42.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:42.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:42.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:42.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:42.512+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:43.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:43.116+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:43.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:43.116+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:43.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:43.116+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:43.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:43.116+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:43.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:43.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:43.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:43.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:43.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:43.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:43.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:43.176+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:43.846 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:43.534+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:43.846 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:43.534+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:43.846 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:43.534+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:43.846 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:43.534+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:44.132 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:34:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:34:43.844+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18176 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:34:44.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:44.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:44.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:44.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:44.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:44.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:44.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:44.132+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:44.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:44.187+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:44.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:44.187+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:44.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:44.187+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:44.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:44.188+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:44.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:44.567+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:44.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:44.567+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:44.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:44.567+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:44.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:44.568+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:45.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:45.120+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:45.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:45.120+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:45.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:45.120+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:45.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:45.121+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:45.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:45.211+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:45.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:45.211+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:45.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:45.211+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:45.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:45.212+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:45.782 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:45.521+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:45.783 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:45.521+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:45.783 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:45.521+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:45.783 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:45.522+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:46.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:46.100+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:46.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:46.100+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:46.100+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:46.100+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:46.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:46.245+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:46.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:46.245+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:46.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:46.245+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:46.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:46.245+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:46.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:46.487+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:46.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:46.488+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:46.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:46.488+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:46.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:46.488+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:47.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:47.234+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:47.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:47.234+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:47.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:47.234+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:47.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:47.235+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:47.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:47.126+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:47.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:47.126+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:47.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:47.126+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:47.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:47.127+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:47.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:47.444+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:47.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:47.444+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:47.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:47.444+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:47.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:47.445+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:48.387 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:48.135+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:48.387 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:48.135+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:48.388 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:48.135+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:48.388 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:48.136+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:48.388 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:48.233+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:48.388 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:48.233+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:48.388 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:48.233+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:48.388 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:48.233+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:48.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:48.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:48.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:48.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:48.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:48.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:48.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:48.493+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:49.126 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:34:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:34:48.846+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18229 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:34:49.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:49.202+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:49.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:49.202+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:49.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:49.202+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:49.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:49.203+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:49.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:49.125+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:49.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:49.125+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:49.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:49.125+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:49.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:49.126+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:49.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:49.453+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:49.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:49.453+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:49.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:49.453+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:49.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:49.454+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:50.411 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:50.096+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:50.412 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:50.096+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:50.412 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:50.096+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:50.412 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:50.097+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:50.412 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:50.233+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:50.413 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:50.233+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:50.413 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:50.233+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:50.413 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:50.233+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:50.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:50.410+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:50.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:50.410+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:50.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:50.410+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:50.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:50.411+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:51.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:51.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:51.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:51.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:51.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:51.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:51.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:51.093+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:51.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:51.262+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:51.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:51.262+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:51.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:51.262+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:51.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:51.262+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:51.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:51.446+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:51.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:51.446+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:51.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:51.446+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:51.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:51.447+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:52.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:52.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:52.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:52.101+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:52.102+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:52.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:52.231+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:52.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:52.231+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:52.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:52.231+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:52.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:52.231+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:52.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:52.486+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:52.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:52.486+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:52.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:52.486+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:52.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:52.487+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:53.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:53.103+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:53.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:53.103+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:53.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:53.103+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:53.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:53.104+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:53.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:53.276+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:53.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:53.276+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:53.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:53.276+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:53.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:53.277+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:53.850 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:53.459+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:53.851 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:53.459+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:53.851 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:53.459+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:53.851 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:53.460+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:54.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:34:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:34:53.848+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18281 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:34:54.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:54.085+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:54.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:54.085+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:54.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:54.085+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:54.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:54.086+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:54.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:54.280+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:54.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:54.280+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:54.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:54.280+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:54.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:54.280+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:54.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:54.509+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:54.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:54.509+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:54.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:54.509+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:54.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:54.510+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:55.253 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:55.050+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:55.254 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:55.050+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:55.254 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:55.050+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:55.254 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:55.050+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:55.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:55.463+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:55.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:55.463+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:55.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:55.463+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:55.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:55.464+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:55.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:55.252+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:55.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:55.252+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:55.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:55.252+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:55.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:55.252+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:56.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:56.060+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:56.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:56.060+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:56.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:56.060+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:56.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:56.061+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:56.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:56.291+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:56.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:56.291+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:56.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:56.291+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:56.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:56.292+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:56.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:56.447+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:56.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:56.447+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:56.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:56.447+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:56.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:56.448+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:57.314 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:57.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:57.314 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:57.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:57.314 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:57.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:57.315 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:57.048+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:57.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:57.312+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:57.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:57.312+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:57.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:57.312+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:57.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:57.313+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:57.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:57.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:57.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:57.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:57.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:57.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:57.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:57.449+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:58.362 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:58.008+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:58.362 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:58.008+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:58.362 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:58.008+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:58.362 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:58.009+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:58.360+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:58.360+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:58.360+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:58.361+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:58.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:58.443+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:58.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:58.443+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:58.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:58.443+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:58.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:58.444+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:59.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:34:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:34:58.850+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18334 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:34:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:59.043+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:59.043+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:59.043+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:34:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:34:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:34:59.045+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:34:59.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:59.487+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:59.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:59.487+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:59.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:59.487+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:34:59.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:34:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:34:59.488+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:34:59.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:59.387+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:59.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:59.387+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:59.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:59.387+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:34:59.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:34:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:34:59.387+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:00.348 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:00.035+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:00.348 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:00.035+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:00.348 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:00.035+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:00.348 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:00.037+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:00.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:00.451+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:00.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:00.451+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:00.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:00.451+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:00.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:00.451+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:00.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:00.346+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:00.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:00.346+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:00.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:00.346+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:00.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:00.347+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:01.391 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:01.054+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:01.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:01.054+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:01.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:01.054+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:01.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:01.055+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:01.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:01.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:01.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:01.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:01.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:01.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:01.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:01.467+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:01.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:01.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:01.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:01.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:01.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:01.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:01.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:01.391+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:02.372 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:02.018+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:02.372 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:02.018+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:02.373 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:02.018+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:02.373 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:02.018+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:02.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:02.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:02.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:02.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:02.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:02.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:02.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:02.466+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:02.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:02.370+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:02.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:02.370+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:02.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:02.370+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:02.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:02.371+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:03.365 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:02.987+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:03.365 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:02.987+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:03.366 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:02.987+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:03.366 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:02.988+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:03.642 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:03.363+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:03.642 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:03.363+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:03.642 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:03.363+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:03.643 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:03.364+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:03.643 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:03.416+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:03.643 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:03.416+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:03.643 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:03.416+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:03.643 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:03.417+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:03.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:35:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:35:03.852+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18386 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:35:04.360 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:04.003+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:04.360 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:04.003+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:04.360 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:04.003+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:04.361 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:04.003+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:04.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:04.406+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:04.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:04.406+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:04.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:04.406+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:04.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:04.408+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:04.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:04.358+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:04.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:04.358+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:04.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:04.358+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:04.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:04.359+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:05.411 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:05.029+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:05.411 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:05.029+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:05.411 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:05.029+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:05.411 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:05.029+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:05.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:05.409+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:05.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:05.409+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:05.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:05.409+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:05.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:05.411+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:05.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:05.409+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:05.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:05.409+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:05.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:05.409+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:05.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:05.410+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:06.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:06.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:06.380 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:06.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:06.380 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:06.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:06.380 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:06.047+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:06.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:06.377+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:06.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:06.377+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:06.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:06.377+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:06.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:06.379+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:06.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:06.436+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:06.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:06.436+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:06.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:06.436+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:06.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:06.437+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:07.403 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:07.005+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:07.403 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:07.005+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:07.403 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:07.005+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:07.403 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:07.006+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:07.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:07.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:07.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:07.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:07.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:07.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:07.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:07.449+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:07.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:07.401+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:07.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:07.401+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:07.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:07.401+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:07.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:07.402+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:08.402 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:08.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:08.402 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:08.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:08.402 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:08.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:08.403 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:08.013+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:08.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:08.403+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:08.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:08.403+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:08.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:08.403+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:08.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:08.403+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:08.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:08.401+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:08.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:08.401+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:08.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:08.401+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:08.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:08.401+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:09.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:35:08.854+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18439 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:35:09.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:08.964+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:09.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:08.964+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:09.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:08.964+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:09.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:08.965+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:09.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:09.368+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:09.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:09.368+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:09.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:09.368+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:09.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:09.369+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:09.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:09.370+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:09.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:09.370+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:09.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:09.370+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:09.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:09.371+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:10.343 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:10.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:10.343 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:10.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:10.344 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:10.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:10.344 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:10.013+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:10.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:10.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:10.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:10.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:10.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:10.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:10.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:10.342+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:10.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:10.357+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:10.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:10.357+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:10.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:10.357+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:10.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:10.358+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:11.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:10.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:11.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:10.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:11.353 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:10.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:11.354 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:10.994+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:11.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:11.369+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:11.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:11.369+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:11.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:11.369+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:11.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:11.370+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:11.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:11.352+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:11.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:11.352+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:11.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:11.352+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:11.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:11.352+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:12.327 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:11.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:12.328 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:11.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:12.328 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:11.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:12.328 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:11.956+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:12.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:12.326+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:12.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:12.326+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:12.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:12.326+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:12.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:12.326+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:12.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:12.364+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:12.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:12.364+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:12.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:12.364+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:12.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:12.366+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:13.365 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:12.991+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:13.366 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:12.991+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:13.366 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:12.991+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:13.367 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:12.992+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:13.643 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:13.364+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:13.643 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:13.364+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:13.643 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:13.364+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:13.643 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:13.364+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:13.643 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:13.408+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:13.644 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:13.408+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:13.644 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:13.408+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:13.644 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:13.409+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:13.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:35:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:35:13.856+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18491 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:35:14.341 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:14.041+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:14.341 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:14.041+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:14.342 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:14.041+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:14.342 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:14.042+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:14.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:14.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:14.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:14.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:14.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:14.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:14.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:14.341+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:14.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:14.431+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:14.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:14.431+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:14.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:14.431+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:14.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:14.432+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:15.311 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:15.054+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:15.311 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:15.054+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:15.312 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:15.054+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:15.312 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:15.055+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:15.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:15.431+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:15.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:15.431+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:15.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:15.431+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:15.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:15.432+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:15.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:15.309+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:15.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:15.309+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:15.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:15.309+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:15.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:15.310+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:16.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:16.087+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:16.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:16.087+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:16.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:16.087+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:16.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:16.088+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:16.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:16.273+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:16.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:16.273+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:16.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:16.273+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:16.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:16.274+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:16.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:16.443+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:16.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:16.443+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:16.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:16.443+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:16.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:16.444+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:17.413 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:17.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:17.413 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:17.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:17.414 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:17.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:17.414 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:17.294+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:17.414 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:17.077+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:17.414 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:17.077+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:17.414 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:17.077+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:17.414 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:17.078+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:17.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:17.411+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:17.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:17.411+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:17.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:17.411+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:17.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:17.412+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:18.332 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:18.055+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:18.332 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:18.055+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:18.332 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:18.055+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:18.332 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:18.056+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:18.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:18.330+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:18.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:18.330+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:18.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:18.330+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:18.632 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:18.330+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:18.632 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:18.456+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:18.632 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:18.456+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:18.633 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:18.456+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:18.633 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:18.457+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:18.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:35:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:35:18.858+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18544 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:35:19.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:19.326+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:19.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:19.326+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:19.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:19.326+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:19.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:19.328+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:19.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:19.086+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:19.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:19.086+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:19.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:19.086+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:19.429 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:19.087+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:19.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:19.423+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:19.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:19.423+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:19.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:19.423+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:19.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:19.424+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:20.361 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:20.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:20.361 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:20.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:20.362 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:20.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:20.362 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:20.040+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:20.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:20.447+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:20.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:20.447+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:20.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:20.447+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:20.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:20.448+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:20.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:20.359+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:20.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:20.359+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:20.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:20.359+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:20.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:20.360+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:21.397 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:21.045+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:21.397 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:21.045+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:21.397 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:21.045+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:21.397 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:21.046+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:21.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:21.422+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:21.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:21.423+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:21.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:21.423+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:21.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:21.424+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:21.678 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:21.394+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:21.678 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:21.394+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:21.678 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:21.394+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:21.678 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:21.395+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:22.361 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:22.034+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:22.361 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:22.034+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:22.361 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:22.034+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:22.361 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:22.035+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:22.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:22.359+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:22.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:22.359+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:22.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:22.359+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:22.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:22.360+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:22.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:22.387+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:22.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:22.387+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:22.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:22.387+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:22.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:22.387+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:23.387 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:23.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:23.387 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:23.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:23.387 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:23.051+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:23.387 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:23.052+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:23.643 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:23.385+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:23.643 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:23.385+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:23.643 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:23.385+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:23.643 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:23.386+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:23.644 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:23.396+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:23.644 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:23.396+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:23.644 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:23.396+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:23.644 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:23.397+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:23.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:35:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:35:23.860+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18596 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:35:24.391 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:24.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:24.391 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:24.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:24.391 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:24.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:24.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:24.077+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:24.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:24.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:24.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:24.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:24.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:24.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:24.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:24.390+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:24.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:24.402+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:24.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:24.402+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:24.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:24.402+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:24.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:24.402+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:25.364 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:25.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:25.364 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:25.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:25.365 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:25.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:25.365 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:25.093+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:25.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:25.362+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:25.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:25.362+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:25.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:25.362+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:25.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:25.363+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:25.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:25.449+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:25.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:25.449+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:25.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:25.449+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:25.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:25.451+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:26.347 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:26.050+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:26.347 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:26.050+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:26.348 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:26.050+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:26.348 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:26.051+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:26.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:26.345+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:26.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:26.345+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:26.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:26.345+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:26.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:26.346+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:26.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:26.481+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:26.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:26.481+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:26.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:26.481+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:26.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:26.481+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:27.356 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:27.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:27.357 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:27.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:27.357 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:27.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:27.357 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:27.063+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:27.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:27.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:27.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:27.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:27.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:27.492+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:27.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:27.493+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:27.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:27.354+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:27.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:27.354+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:27.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:27.354+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:27.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:27.355+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:28.375 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:28.102+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:28.376 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:28.102+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:28.376 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:28.102+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:28.376 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:28.103+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:28.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:28.374+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:28.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:28.374+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:28.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:28.374+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:28.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:28.374+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:28.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:28.534+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:28.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:28.534+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:28.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:28.534+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:28.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:28.535+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:29.150 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:35:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:35:28.862+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18649 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:35:29.414 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:29.148+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:29.415 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:29.148+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:29.415 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:29.148+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:29.415 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:29.149+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:29.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:29.413+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:29.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:29.413+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:29.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:29.413+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:29.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:29.414+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:29.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:29.551+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:29.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:29.551+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:29.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:29.551+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:29.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:29.553+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:30.581 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:30.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:30.582 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:30.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:30.582 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:30.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:30.582 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:30.408+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:30.582 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:30.176+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:30.582 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:30.176+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:30.583 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:30.176+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:30.583 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:30.177+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:30.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:30.580+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:30.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:30.580+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:30.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:30.580+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:30.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:30.580+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:31.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:31.195+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:31.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:31.195+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:31.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:31.195+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:31.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:31.196+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:31.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:31.379+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:31.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:31.379+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:31.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:31.379+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:31.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:31.379+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:31.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:31.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:31.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:31.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:31.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:31.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:31.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:31.595+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:32.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:32.364+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:32.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:32.364+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:32.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:32.364+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:32.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:32.365+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:32.570 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:32.209+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:32.570 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:32.209+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:32.570 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:32.209+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:32.570 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:32.210+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:32.567+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:32.567+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:32.567+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:32.568+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:33.605 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:33.193+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:33.605 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:33.193+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:33.606 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:33.193+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:33.606 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:33.193+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:33.606 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:33.355+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:33.606 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:33.355+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:33.606 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:33.355+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.711923+0000 (oldest deadline 2024-04-27T04:25:23.711923+0000) 2024-04-27T04:35:33.606 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:33.356+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:33.866 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:33.603+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:33.866 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:33.603+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:33.866 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:33.603+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:24:58.975953+0000 (oldest deadline 2024-04-27T04:25:18.975953+0000) 2024-04-27T04:35:33.866 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:33.604+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:34.157 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:35:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:35:33.864+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18701 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:35:34.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:34.366+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:34.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:34.156+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:34.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:34.156+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:34.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:34.156+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:34.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:34.157+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:34.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:34.608+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:35.367 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:35.109+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:35.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:35.109+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:35.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:35.109+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:25:03.736900+0000 (oldest deadline 2024-04-27T04:25:23.736900+0000) 2024-04-27T04:35:35.368 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:35.109+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:35.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:35.366+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:35.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:35.568+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:36.403 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:36.111+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:36.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:36.401+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:36.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:36.586+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:37.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:37.371+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:37.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:37.156+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:37.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:37.631+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:38.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:38.163+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:38.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:38.343+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:38.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:35:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:35:38.866+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18754 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:35:38.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:38.646+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:39.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:39.132+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:39.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:39.366+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:40.092 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:39.684+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:40.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:40.091+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:40.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:40.333+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:41.113 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:40.716+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:41.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:41.336+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:41.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:41.111+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:41.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:41.679+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:42.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:42.101+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:42.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:42.330+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:42.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:42.667+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:43.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:43.127+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:43.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:43.348+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:43.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:35:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:35:43.867+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18806 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:35:43.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:43.644+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:44.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:44.097+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:44.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:44.309+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:44.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:44.667+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:45.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:45.114+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:45.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:45.292+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:46.068 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:45.673+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:46.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:46.066+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:46.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:46.270+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:47.038 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:46.723+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:47.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:47.037+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:47.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:47.252+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:48.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:47.992+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:48.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:47.770+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:48.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:48.270+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:48.996 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:35:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:35:48.869+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18859 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:35:48.997 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:48.724+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:49.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:48.995+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:49.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:49.241+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:49.992 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:49.734+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:50.285 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:49.991+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:50.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:50.284+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:51.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:50.965+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:51.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:50.741+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:51.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:51.333+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:52.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:51.772+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:52.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:51.920+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:52.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:52.283+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:53.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:52.790+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:53.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:52.964+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:53.645 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:53.328+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:53.909 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:53.746+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:53.909 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:35:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:35:53.871+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18911 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:35:54.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:53.923+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:54.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:54.345+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:55.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:54.911+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:55.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:54.730+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:55.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:55.357+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:35:55.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:55.357+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:35:55.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:55.357+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:35:55.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:55.358+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:56.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:55.872+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:56.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:55.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:35:56.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:55.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:35:56.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:55.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:35:56.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:55.732+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:56.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:56.339+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:35:56.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:56.339+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:35:56.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:56.339+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:35:56.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:56.340+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:57.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:56.854+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:57.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:56.702+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:35:57.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:56.702+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:35:57.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:56.702+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:35:57.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:56.703+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:57.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:57.346+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:35:57.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:57.346+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:35:57.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:57.346+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:35:57.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:57.347+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:58.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:57.844+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:58.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:57.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:35:58.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:57.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:35:58.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:57.692+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:35:58.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:57.693+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:58.304+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:35:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:58.304+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:35:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:58.304+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:35:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:58.305+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:59.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:35:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:35:58.872+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 18964 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:35:59.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:58.740+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:35:59.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:58.740+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:35:59.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:58.740+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:35:59.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:58.742+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:35:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:58.853+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:35:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:58.853+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:35:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:58.853+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:35:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:58.853+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:35:59.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:59.329+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:35:59.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:59.329+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:35:59.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:59.329+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:35:59.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:35:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:35:59.330+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:00.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:59.735+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:00.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:59.735+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:00.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:59.735+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:00.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:35:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:35:59.736+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:00.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:59.834+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:00.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:59.834+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:00.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:59.834+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:00.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:35:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:35:59.835+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:00.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:00.348+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:00.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:00.348+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:00.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:00.348+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:00.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:00.348+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:01.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:00.816+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:01.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:00.816+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:01.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:00.816+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:01.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:00.817+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:01.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:00.687+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:01.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:00.687+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:01.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:00.687+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:01.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:00.688+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:01.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:01.333+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:01.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:01.333+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:01.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:01.333+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:01.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:01.334+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:01.846 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:01.642+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:01.846 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:01.642+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:01.846 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:01.642+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:01.847 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:01.643+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:02.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:01.843+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:02.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:01.843+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:02.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:01.843+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:02.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:01.844+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:02.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:02.303+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:02.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:02.303+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:02.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:02.303+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:02.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:02.303+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:03.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:02.818+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:03.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:02.818+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:03.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:02.818+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:03.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:02.819+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:03.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:02.687+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:03.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:02.687+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:03.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:02.687+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:03.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:02.688+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:03.646 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:03.275+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:03.646 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:03.275+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:03.646 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:03.275+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:03.646 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:03.276+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:03.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:36:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:36:03.874+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19016 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:36:03.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:03.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:03.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:03.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:03.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:03.774+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:03.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:03.775+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:03.691+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:03.691+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:03.691+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:03.692+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:04.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:04.326+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:04.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:04.326+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:04.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:04.326+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:04.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:04.326+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:05.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:04.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:05.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:04.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:05.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:04.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:05.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:04.701+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:05.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:04.750+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:05.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:04.750+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:05.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:04.750+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:05.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:04.751+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:05.655 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:05.299+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:05.655 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:05.299+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:05.655 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:05.299+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:05.655 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:05.300+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:05.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:05.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:05.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:05.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:05.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:05.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:05.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:05.705+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:05.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:05.731+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:05.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:05.731+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:05.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:05.731+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:05.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:05.733+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:06.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:06.319+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:06.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:06.319+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:06.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:06.319+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:06.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:06.320+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:07.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:06.741+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:07.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:06.741+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:07.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:06.741+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:07.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:06.742+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:07.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:06.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:07.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:06.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:07.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:06.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:07.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:06.686+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:07.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:07.311+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:07.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:07.311+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:07.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:07.311+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:07.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:07.311+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:08.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:07.713+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:08.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:07.713+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:08.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:07.713+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:08.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:07.714+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:08.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:07.746+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:08.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:07.747+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:08.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:07.747+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:08.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:07.747+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:08.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:08.300+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:08.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:08.300+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:08.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:08.300+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:08.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:08.300+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:09.031 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:36:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:36:08.877+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19069 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:36:09.031 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:08.757+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:09.031 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:08.757+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:09.031 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:08.757+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:09.032 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:08.758+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:09.032 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:08.744+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:09.032 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:08.744+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:09.032 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:08.744+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:09.032 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:08.745+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:09.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:09.252+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:09.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:09.252+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:09.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:09.252+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:09.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:09.253+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:10.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:09.754+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:10.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:09.754+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:10.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:09.754+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:10.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:09.755+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:10.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:09.778+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:10.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:09.778+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:10.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:09.778+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:10.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:09.779+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:10.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:10.213+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:10.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:10.213+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:10.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:10.213+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:10.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:10.214+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:11.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:10.788+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:11.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:10.788+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:11.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:10.788+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:11.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:10.789+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:11.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:10.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:11.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:10.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:11.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:10.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:11.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:10.821+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:11.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:11.205+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:11.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:11.205+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:11.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:11.205+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:11.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:11.206+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:12.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:11.822+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:12.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:11.822+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:12.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:11.822+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:12.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:11.823+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:12.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:11.852+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:12.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:11.852+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:12.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:11.852+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:12.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:11.852+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:12.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:12.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:12.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:12.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:12.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:12.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:12.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:12.222+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:13.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:12.866+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:13.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:12.866+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:13.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:12.866+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:13.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:12.868+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:13.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:12.857+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:13.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:12.857+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:13.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:12.857+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:13.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:12.859+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:13.647 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:13.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:13.647 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:13.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:13.647 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:13.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:13.647 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:13.204+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:13.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:36:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:36:13.879+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19121 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:36:13.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:13.886+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:13.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:13.886+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:13.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:13.886+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:13.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:13.888+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:13.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:13.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:13.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:13.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:13.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:13.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:13.926 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:13.820+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:14.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:14.230+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:14.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:14.230+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:14.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:14.230+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:14.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:14.231+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:15.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:14.796+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:15.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:14.796+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:15.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:14.796+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:15.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:14.797+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:15.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:14.889+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:15.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:14.889+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:15.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:14.889+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:15.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:14.890+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:15.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:15.257+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:15.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:15.257+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:15.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:15.257+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:15.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:15.258+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:16.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:15.929+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:16.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:15.929+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:16.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:15.929+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:16.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:15.930+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:16.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:15.812+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:16.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:15.812+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:16.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:15.812+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:16.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:15.812+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:16.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:16.228+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:16.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:16.228+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:16.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:16.228+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:16.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:16.229+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:17.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:16.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:17.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:16.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:17.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:16.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:17.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:16.978+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:17.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:16.788+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:17.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:16.788+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:17.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:16.788+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:17.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:16.789+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:17.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:17.234+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:17.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:17.234+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:17.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:17.234+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:17.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:17.234+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:18.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:17.946+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:18.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:17.946+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:18.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:17.946+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:18.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:17.947+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:18.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:17.752+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:18.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:17.752+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:18.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:17.752+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:18.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:17.753+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:18.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:18.273+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:18.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:18.273+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:18.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:18.273+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:18.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:18.273+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:19.111 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:36:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:36:18.881+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19174 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:36:19.111 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:18.924+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:19.111 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:18.924+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:19.111 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:18.924+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:19.112 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:18.925+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:19.112 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:18.780+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:19.112 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:18.780+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:19.112 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:18.780+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:19.112 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:18.781+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:19.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:19.322+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:19.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:19.322+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:19.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:19.322+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:19.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:19.323+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:20.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:19.937+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:20.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:19.938+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:20.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:19.938+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:20.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:19.938+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:20.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:19.821+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:20.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:19.821+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:20.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:19.821+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:20.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:19.821+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:20.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:20.333+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:20.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:20.333+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:20.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:20.333+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:20.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:20.334+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:21.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:20.910+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:21.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:20.910+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:21.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:20.910+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:21.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:20.911+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:21.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:20.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:21.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:20.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:21.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:20.845+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:21.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:20.847+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:21.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:21.315+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:21.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:21.315+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:21.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:21.315+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:21.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:21.316+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:22.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:21.882+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:22.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:21.882+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:22.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:21.882+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:22.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:21.883+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:22.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:21.888+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:22.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:21.888+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:22.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:21.888+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:22.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:21.889+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:22.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:22.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:22.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:22.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:22.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:22.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:22.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:22.309+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:23.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:22.850+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:23.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:22.850+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:23.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:22.850+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:23.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:22.851+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:23.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:22.867+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:23.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:22.867+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:23.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:22.867+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:23.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:22.868+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:23.648 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:23.293+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:23.648 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:23.293+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:23.648 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:23.293+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:23.648 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:23.294+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:23.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:36:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:36:23.883+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19226 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:36:23.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:23.818+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:23.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:23.818+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:23.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:23.818+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:23.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:23.820+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:23.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:23.841+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:23.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:23.841+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:23.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:23.841+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:23.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:23.842+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:24.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:24.262+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:24.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:24.262+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:24.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:24.262+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:24.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:24.262+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:25.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:24.770+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:25.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:24.770+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:25.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:24.770+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:25.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:24.771+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:25.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:24.828+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:25.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:24.828+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:25.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:24.828+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:25.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:24.829+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:25.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:25.216+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:25.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:25.216+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:25.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:25.216+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:25.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:25.217+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:26.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:25.777+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:26.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:25.777+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:26.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:25.777+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:26.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:25.778+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:26.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:25.870+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:26.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:25.870+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:26.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:25.870+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:26.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:25.871+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:26.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:26.247+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:26.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:26.247+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:26.677 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:26.247+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:26.678 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:26.247+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:27.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:26.824+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:27.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:26.824+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:27.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:26.824+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:27.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:26.825+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:27.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:26.871+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:27.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:26.871+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:27.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:26.871+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:27.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:26.872+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:27.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:27.270+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:27.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:27.270+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:27.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:27.270+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:27.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:27.270+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:28.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:27.803+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:28.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:27.803+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:28.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:27.803+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:28.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:27.804+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:28.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:27.864+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:28.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:27.864+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:28.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:27.864+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:28.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:27.865+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:28.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:28.226+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:28.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:28.226+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:28.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:28.226+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:28.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:28.226+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:29.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:36:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:36:28.884+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19279 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:36:29.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:28.832+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:29.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:28.832+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:29.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:28.832+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:29.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:28.834+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:29.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:28.881+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:29.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:28.881+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:29.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:28.881+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:29.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:28.882+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:29.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:29.250+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:29.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:29.250+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:29.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:29.250+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:29.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:29.250+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:30.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:29.858+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:30.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:29.858+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:30.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:29.858+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:30.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:29.859+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:30.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:29.834+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:30.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:29.834+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:30.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:29.834+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:30.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:29.834+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:30.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:30.266+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:30.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:30.266+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:30.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:30.266+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:30.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:30.267+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:31.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:30.817+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:31.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:30.817+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:31.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:30.817+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:31.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:30.818+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:31.177 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:30.831+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:31.178 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:30.831+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:31.178 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:30.831+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:31.178 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:30.832+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:31.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:31.296+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:31.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:31.296+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:31.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:31.296+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:31.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:31.296+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:32.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:31.829+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:32.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:31.829+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:32.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:31.829+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:32.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:31.829+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:32.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:31.871+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:32.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:31.871+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:32.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:31.871+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:32.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:31.871+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:32.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:32.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:32.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:32.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:32.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:32.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:32.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:32.272+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:33.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:32.838+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:33.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:32.838+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:33.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:32.838+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:33.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:32.839+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:33.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:32.797+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:33.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:32.797+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:33.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:32.797+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:33.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:32.798+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:33.648 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:33.228+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:33.648 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:33.228+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:33.648 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:33.228+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:33.648 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:33.229+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:33.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:36:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:36:33.886+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19331 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:36:33.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:33.888+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:33.888+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:33.888+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:33.889+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:33.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:33.837+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:33.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:33.837+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:33.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:33.837+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:33.926 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:33.838+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:34.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:34.215+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:34.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:34.215+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:34.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:34.215+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:34.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:34.215+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:35.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:34.827+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:35.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:34.827+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:35.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:34.827+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:35.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:34.828+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:35.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:34.898+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:35.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:34.898+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:35.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:34.898+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:35.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:34.899+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:35.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:35.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:35.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:35.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:35.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:35.203+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:35.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:35.203+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:36.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:35.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:36.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:35.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:36.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:35.823+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:36.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:35.824+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:36.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:35.876+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:36.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:35.876+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:36.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:35.876+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:36.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:35.876+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:36.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:36.252+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:36.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:36.252+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:36.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:36.252+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:36.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:36.253+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:37.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:36.813+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:37.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:36.813+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:37.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:36.813+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:37.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:36.814+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:37.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:36.828+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:37.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:36.828+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:37.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:36.828+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:37.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:36.829+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:37.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:37.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:37.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:37.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:37.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:37.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:37.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:37.295+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:38.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:37.778+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:38.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:37.778+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:38.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:37.778+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:38.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:37.779+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:38.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:37.825+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:38.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:37.825+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:38.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:37.825+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:38.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:37.826+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:38.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:38.325+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:38.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:38.325+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:38.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:38.325+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:38.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:38.325+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:39.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:36:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:36:38.888+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19384 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:36:39.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:38.809+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:39.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:38.809+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:39.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:38.809+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:39.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:38.810+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:39.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:38.794+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:39.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:38.794+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:39.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:38.794+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:39.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:38.795+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:39.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:39.287+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:39.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:39.287+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:39.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:39.287+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:39.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:39.288+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:40.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:39.765+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:40.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:39.765+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:40.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:39.765+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:40.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:39.766+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:40.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:39.769+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:40.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:39.769+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:40.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:39.769+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:40.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:39.770+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:40.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:40.244+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:40.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:40.244+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:40.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:40.244+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:40.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:40.244+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:41.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:40.807+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:41.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:40.807+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:41.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:40.807+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:41.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:40.808+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:41.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:40.724+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:41.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:40.724+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:41.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:40.724+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:41.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:40.725+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:41.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:41.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:41.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:41.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:41.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:41.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:41.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:41.217+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:41.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:41.793+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:41.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:41.793+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:41.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:41.793+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:41.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:41.794+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:41.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:41.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:41.760+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:41.761+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:42.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:42.231+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:42.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:42.231+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:42.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:42.231+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:42.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:42.232+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:43.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:42.760+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:43.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:42.760+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:43.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:42.760+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:43.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:42.761+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:43.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:42.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:43.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:42.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:43.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:42.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:43.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:42.734+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:43.648 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:43.223+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:43.648 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:43.223+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:43.648 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:43.223+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:43.648 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:43.224+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:43.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:36:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:36:43.890+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19436 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:36:43.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:43.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:43.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:43.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:43.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:43.805+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:43.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:43.806+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:43.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:43.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:43.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:43.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:43.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:43.707+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:43.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:43.708+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:44.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:44.256+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:44.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:44.256+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:44.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:44.256+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:44.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:44.257+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:45.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:44.837+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:45.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:44.837+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:45.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:44.837+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:45.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:44.838+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:45.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:44.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:45.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:44.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:45.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:44.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:45.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:44.705+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:45.592 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:45.270+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:45.592 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:45.270+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:45.592 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:45.270+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:45.593 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:45.271+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:45.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:45.801+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:45.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:45.801+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:45.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:45.801+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:45.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:45.802+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:45.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:45.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:45.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:45.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:45.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:45.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:45.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:45.698+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:46.652 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:46.320+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:46.652 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:46.320+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:46.653 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:46.320+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:46.653 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:46.320+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:46.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:46.773+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:46.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:46.773+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:46.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:46.773+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:46.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:46.774+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:46.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:46.650+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:46.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:46.650+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:46.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:46.650+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:46.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:46.652+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:47.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:47.363+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:47.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:47.363+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:47.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:47.363+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:47.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:47.364+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:48.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:47.739+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:48.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:47.739+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:48.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:47.739+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:48.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:47.741+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:48.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:47.674+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:48.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:47.674+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:48.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:47.674+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:48.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:47.674+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:48.654 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:48.402+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:48.654 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:48.402+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:48.654 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:48.402+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:48.654 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:48.402+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:48.924 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:36:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:36:48.892+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19489 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:36:48.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:48.652+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:48.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:48.652+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:48.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:48.652+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:48.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:48.653+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:48.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:48.709+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:48.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:48.709+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:48.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:48.709+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:48.926 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:48.710+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:49.628 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:49.420+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:49.629 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:49.420+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:49.629 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:49.420+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:49.629 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:49.421+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:49.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:49.758+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:49.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:49.758+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:49.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:49.758+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:49.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:49.759+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:49.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:49.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:49.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:49.627+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:50.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:50.639+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:50.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:50.639+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:50.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:50.639+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:50.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:50.639+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:50.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:50.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:50.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:50.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:50.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:50.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:50.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:50.407+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:51.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:50.807+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:51.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:50.807+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:51.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:50.807+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:51.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:50.808+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:51.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:51.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:51.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:51.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:51.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:51.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:51.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:51.612+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:51.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:51.419+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:51.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:51.419+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:51.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:51.419+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:51.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:51.420+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:52.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:51.785+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:52.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:51.785+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:52.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:51.785+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:52.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:51.786+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:52.743 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:52.424+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:52.744 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:52.424+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:52.744 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:52.424+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:52.744 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:52.425+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:52.744 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:52.583+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:52.744 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:52.583+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:52.744 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:52.583+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:52.745 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:52.583+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:53.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:52.741+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:53.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:52.742+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:53.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:52.742+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:53.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:52.742+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:53.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:53.404+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:53.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:53.404+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:53.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:53.404+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:53.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:53.404+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:53.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:53.562+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:53.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:53.562+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:53.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:53.562+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:53.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:53.563+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:54.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:36:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:36:53.894+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19541 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:36:54.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:53.697+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:54.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:53.697+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:54.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:53.697+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:54.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:53.697+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:54.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:54.373+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:54.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:54.373+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:54.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:54.373+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:54.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:54.374+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:54.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:54.543+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:54.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:54.544+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:54.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:54.544+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:54.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:54.544+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:55.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:54.690+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:55.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:54.690+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:55.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:54.690+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:55.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:54.691+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:55.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:55.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:55.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:55.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:55.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:55.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:55.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:55.542+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:55.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:55.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:55.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:55.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:55.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:55.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:55.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:55.342+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:56.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:55.718+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:56.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:55.718+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:56.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:55.718+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:56.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:55.718+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:56.586 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:56.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:56.586 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:56.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:56.586 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:56.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:56.586 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:56.309+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:56.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:56.703+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:56.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:56.703+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:56.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:56.703+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:56.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:56.703+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:56.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:56.584+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:56.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:56.584+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:56.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:56.584+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:56.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:56.584+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:57.568 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:57.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:57.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:57.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:57.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:57.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:57.569 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:57.312+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:57.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:57.727+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:57.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:57.727+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:57.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:57.727+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:57.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:57.727+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:57.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:57.566+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:57.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:57.566+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:57.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:57.566+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:57.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:57.568+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:58.552 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:58.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:58.552 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:58.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:58.552 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:58.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:58.552 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:58.295+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:58.898 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:58.687+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:58.899 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:58.687+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:58.899 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:58.687+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:58.899 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:58.688+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:58.899 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:58.550+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:58.899 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:58.550+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:58.899 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:58.550+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:58.900 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:58.551+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:59.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:36:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:36:58.896+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19594 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:36:59.578 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:59.266+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:59.579 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:59.266+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:59.579 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:59.266+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:36:59.579 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:36:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:36:59.267+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:36:59.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:59.576+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:59.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:59.576+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:59.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:59.576+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:36:59.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:36:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:36:59.577+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:36:59.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:59.736+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:59.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:59.736+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:59.926 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:59.736+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:36:59.926 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:36:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:36:59.736+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:00.539 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:00.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:00.540 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:00.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:00.540 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:00.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:00.540 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:00.222+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:00.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:00.692+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:00.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:00.692+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:00.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:00.692+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:00.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:00.693+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:00.537+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:00.537+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:00.537+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:00.538+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:01.529 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:01.255+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:01.529 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:01.255+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:01.529 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:01.255+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:01.530 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:01.256+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:01.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:01.687+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:01.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:01.687+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:01.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:01.687+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:01.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:01.688+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:01.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:01.526+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:01.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:01.526+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:01.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:01.526+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:01.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:01.528+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:02.555 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:02.232+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:02.555 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:02.232+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:02.555 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:02.232+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:02.556 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:02.234+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:02.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:02.700+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:02.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:02.700+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:02.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:02.700+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:02.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:02.701+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:02.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:02.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:02.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:02.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:02.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:02.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:02.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:02.554+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:03.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:03.255+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:03.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:03.255+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:03.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:03.255+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:03.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:03.256+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:03.900 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:03.714+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:03.900 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:03.714+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:03.900 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:03.714+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:03.900 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:03.715+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:03.900 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:03.569+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:03.901 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:03.569+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:03.901 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:03.569+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:03.901 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:03.570+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:04.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:37:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:37:03.898+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19646 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:37:04.539 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:04.251+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:04.539 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:04.251+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:04.539 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:04.251+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:04.539 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:04.252+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:04.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:04.758+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:04.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:04.758+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:04.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:04.758+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:04.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:04.760+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:04.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:04.537+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:04.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:04.537+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:04.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:04.537+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:04.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:04.538+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:05.531 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:05.235+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:05.532 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:05.235+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:05.532 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:05.235+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:05.532 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:05.235+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:05.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:05.767+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:05.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:05.767+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:05.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:05.767+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:05.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:05.767+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:05.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:05.530+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:05.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:05.530+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:05.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:05.530+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:05.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:05.531+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:06.543 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:06.234+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:06.544 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:06.234+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:06.544 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:06.234+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:06.544 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:06.234+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:06.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:06.745+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:06.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:06.745+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:06.924 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:06.745+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:06.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:06.746+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:06.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:06.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:06.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:06.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:06.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:06.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:06.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:06.542+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:07.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:07.204+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:07.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:07.204+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:07.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:07.204+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:07.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:07.205+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:07.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:07.571+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:07.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:07.571+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:07.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:07.571+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:07.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:07.572+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:07.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:07.735+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:07.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:07.735+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:07.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:07.735+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:07.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:07.736+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:08.526 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:08.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:08.526 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:08.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:08.526 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:08.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:08.526 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:08.222+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:08.784 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:08.524+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:08.784 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:08.524+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:08.785 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:08.524+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:08.785 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:08.525+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:09.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:37:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:37:08.900+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19699 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:37:09.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:08.782+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:09.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:08.782+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:09.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:08.782+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:09.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:08.783+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:09.509 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:09.181+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:09.510 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:09.181+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:09.510 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:09.181+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:09.510 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:09.182+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:09.786 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:09.520+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:09.787 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:09.520+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:09.787 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:09.520+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:09.787 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:09.521+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:10.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:09.784+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:10.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:09.784+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:10.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:09.784+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:10.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:09.786+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:10.518 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:10.207+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:10.519 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:10.207+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:10.519 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:10.207+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:10.519 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:10.208+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:10.821 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:10.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:10.822 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:10.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:10.822 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:10.516+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:10.822 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:10.518+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:11.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:10.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:11.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:10.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:11.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:10.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:11.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:10.820+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:11.513 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:11.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:11.513 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:11.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:11.513 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:11.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:11.513 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:11.199+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:11.793 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:11.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:11.793 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:11.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:11.794 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:11.511+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:11.794 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:11.512+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:12.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:11.791+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:12.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:11.791+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:12.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:11.791+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:12.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:11.792+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:12.543 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:12.243+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:12.543 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:12.243+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:12.543 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:12.243+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:12.544 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:12.243+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:12.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:12.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:12.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:12.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:12.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:12.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:12.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:12.542+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:12.926 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:12.786+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:12.926 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:12.786+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:12.927 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:12.786+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:12.927 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:12.787+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:13.585 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:13.248+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:13.586 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:13.248+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:13.586 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:13.248+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:13.586 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:13.249+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:13.904 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:13.831+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:13.904 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:13.831+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:13.904 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:13.831+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:13.904 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:13.831+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:13.904 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:13.583+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:13.905 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:13.583+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:13.905 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:13.583+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:13.905 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:13.584+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:14.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:37:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:37:13.901+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19751 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:37:14.552 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:14.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:14.552 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:14.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:14.552 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:14.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:14.553 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:14.272+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:14.840 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:14.569+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:14.841 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:14.569+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:14.841 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:14.569+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:14.841 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:14.570+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:15.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:14.838+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:15.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:14.838+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:15.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:14.838+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:15.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:14.840+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:15.565 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:15.227+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:15.566 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:15.227+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:15.566 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:15.227+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:15.566 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:15.227+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:15.824 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:15.563+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:15.824 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:15.563+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:15.824 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:15.563+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:15.824 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:15.564+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:16.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:15.822+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:16.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:15.822+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:16.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:15.822+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:16.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:15.823+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:16.543 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:16.271+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:16.543 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:16.271+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:16.543 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:16.271+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:16.543 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:16.272+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:16.802 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:16.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:16.802 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:16.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:16.802 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:16.541+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:16.802 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:16.542+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:17.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:16.799+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:17.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:16.799+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:17.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:16.799+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:17.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:16.800+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:17.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:17.507+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:17.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:17.507+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:17.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:17.507+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:17.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:17.508+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:17.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:17.300+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:17.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:17.300+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:17.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:17.300+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:17.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:17.300+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:18.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:17.808+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:18.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:17.808+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:18.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:17.808+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:18.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:17.810+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:18.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:18.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:18.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:18.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:18.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:18.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:18.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:18.341+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:18.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:18.538+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:18.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:18.538+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:18.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:18.538+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:18.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:18.540+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:19.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:37:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:37:18.903+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19804 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:37:19.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:18.847+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:19.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:18.847+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:19.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:18.847+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:19.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:18.848+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:19.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:19.505+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:19.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:19.505+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:19.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:19.505+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:19.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:19.506+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:19.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:19.343+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:19.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:19.343+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:19.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:19.343+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:19.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:19.343+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:20.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:19.887+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:20.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:19.887+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:20.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:19.887+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:20.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:19.887+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:20.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:20.520+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:20.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:20.520+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:20.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:20.520+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:20.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:20.521+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:20.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:20.324+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:20.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:20.324+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:20.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:20.324+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:20.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:20.325+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:21.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:20.874+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:21.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:20.874+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:21.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:20.874+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:21.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:20.875+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:21.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:21.501+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:21.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:21.501+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:21.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:21.501+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:21.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:21.503+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:21.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:21.323+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:21.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:21.323+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:21.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:21.323+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:21.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:21.324+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:22.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:21.852+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:22.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:21.852+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:22.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:21.852+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:22.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:21.853+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:22.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:22.338+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:22.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:22.338+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:22.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:22.338+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:22.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:22.339+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:22.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:22.512+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:22.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:22.512+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:22.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:22.512+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:22.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:22.513+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:23.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:22.812+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:23.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:22.812+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:23.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:22.812+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:23.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:22.813+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:23.651 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:23.369+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:23.651 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:23.369+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:23.651 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:23.369+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:23.651 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:23.369+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:23.652 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:23.475+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:23.652 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:23.475+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:23.652 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:23.475+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:23.652 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:23.476+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:23.907 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:23.860+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:23.908 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:23.860+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:23.908 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:23.860+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:23.908 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:23.861+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:24.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:37:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:37:23.905+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19856 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:37:24.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:24.450+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:24.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:24.450+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:24.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:24.450+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:24.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:24.450+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:24.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:24.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:24.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:24.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:24.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:24.389+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:24.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:24.390+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:25.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:24.894+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:25.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:24.894+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:25.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:24.894+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:25.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:24.894+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:25.902 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:25.483+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:25.902 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:25.483+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:25.902 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:25.483+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:25.903 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:25.483+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:25.903 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:25.432+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:25.903 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:25.432+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:25.903 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:25.432+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:25.903 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:25.433+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:26.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:25.900+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:26.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:25.900+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:26.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:25.900+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:26.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:25.901+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:26.872 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:26.497+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:26.873 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:26.497+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:26.873 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:26.497+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:26.873 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:26.498+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:26.873 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:26.433+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:26.873 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:26.433+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:26.874 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:26.433+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:26.874 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:26.433+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:27.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:26.870+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:27.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:26.870+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:27.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:26.870+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:27.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:26.871+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:27.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:27.453+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:27.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:27.453+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:27.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:27.453+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:27.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:27.454+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:27.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:27.451+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:27.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:27.451+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:27.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:27.451+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:27.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:27.452+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:28.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:27.919+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:28.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:27.919+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:28.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:27.919+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:28.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:27.920+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:28.910 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:28.442+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:28.910 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:28.442+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:28.910 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:28.442+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:28.910 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:28.442+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:28.911 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:28.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:28.911 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:28.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:28.911 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:28.464+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:28.911 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:28.465+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:29.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:37:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:37:28.907+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19909 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:37:29.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:28.960+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:29.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:28.960+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:29.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:28.960+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:29.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:28.962+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:29.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:29.512+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:29.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:29.512+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:29.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:29.512+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:29.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:29.514+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:29.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:29.448+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:29.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:29.448+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:29.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:29.448+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:29.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:29.448+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:30.419 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:29.970+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:30.419 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:29.970+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:30.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:29.970+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:30.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:29.971+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:30.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:30.559+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:30.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:30.559+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:30.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:30.559+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:30.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:30.560+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:30.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:30.416+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:30.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:30.416+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:30.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:30.416+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:30.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:30.417+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:31.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:30.966+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:31.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:30.966+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:31.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:30.966+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:31.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:30.967+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:31.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:31.448+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:31.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:31.448+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:31.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:31.448+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:31.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:31.449+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:31.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:31.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:31.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:31.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:31.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:31.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:31.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:31.594+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:32.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:31.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:32.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:31.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:32.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:31.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:32.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:31.974+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:32.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:32.422+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:32.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:32.422+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:32.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:32.422+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:32.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:32.422+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:32.577+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:32.577+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:32.577+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:32.578+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:33.405 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:32.966+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:33.405 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:32.966+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:33.405 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:32.966+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:33.405 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:32.967+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:33.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:33.402+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:33.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:33.402+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:33.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:33.402+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:33.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:33.402+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:33.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:33.622+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:33.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:33.622+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:33.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:33.622+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:33.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:33.624+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:34.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:37:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:37:33.909+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19954 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:37:34.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:34.001+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:34.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:34.001+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:34.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:34.001+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:34.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:34.002+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:34.713 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:34.599+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:34.714 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:34.599+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:34.714 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:34.599+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:34.714 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:34.600+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:34.715 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:34.423+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:34.715 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:34.423+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:34.715 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:34.423+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:34.716 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:34.424+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:35.420 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:34.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:35.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:34.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:35.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:34.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:35.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:34.976+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:35.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:35.417+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:35.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:35.417+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:35.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:35.417+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:35.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:35.418+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:35.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:35.570+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:35.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:35.570+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:35.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:35.570+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:35.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:35.571+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:36.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:35.934+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:36.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:35.934+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:36.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:35.934+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:36.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:35.934+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:36.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:36.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:36.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:36.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:36.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:36.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:36.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:36.595+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:36.927 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:36.437+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:36.927 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:36.438+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:36.927 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:36.438+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:36.927 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:36.438+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:37.399 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:36.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:37.399 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:36.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:37.399 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:36.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:37.400 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:36.975+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:37.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:37.560+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:37.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:37.561+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:37.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:37.561+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:37.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:37.561+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:37.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:37.396+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:37.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:37.396+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:37.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:37.396+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:37.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:37.397+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:38.362 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:37.970+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:38.363 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:37.970+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:38.363 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:37.970+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:38.363 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:37.971+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:38.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:38.543+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:38.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:38.543+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:38.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:38.543+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:38.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:38.544+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:38.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:38.359+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:38.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:38.359+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:38.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:38.359+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:38.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:38.360+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:39.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:37:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:37:38.911+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 19997 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:37:39.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:38.961+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:39.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:38.961+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:39.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:38.961+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:39.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:38.961+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:39.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:39.410+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:39.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:39.410+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:39.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:39.410+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:39.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:39.410+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:39.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:39.521+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:39.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:39.521+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:39.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:39.521+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:39.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:39.522+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:40.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:39.931+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:40.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:39.931+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:40.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:39.931+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:40.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:39.931+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:40.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:40.504+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:40.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:40.504+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:40.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:40.504+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:40.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:40.505+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:40.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:40.374+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:40.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:40.374+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:40.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:40.374+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:40.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:40.375+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:41.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:40.890+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:41.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:40.890+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:41.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:40.890+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:41.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:40.891+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:41.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:41.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:41.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:41.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:41.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:41.390+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:41.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:41.390+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:41.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:41.473+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:41.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:41.473+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:41.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:41.473+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:41.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:41.473+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:42.355 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:41.929+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:42.355 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:41.929+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:42.355 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:41.929+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:42.355 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:41.930+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:42.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:42.496+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:42.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:42.496+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:42.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:42.496+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:42.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:42.497+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:42.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:42.353+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:42.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:42.353+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:42.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:42.353+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:42.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:42.354+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:43.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:42.914+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:43.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:42.914+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:43.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:42.914+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:43.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:42.915+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:43.652 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:43.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:43.652 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:43.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:43.652 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:43.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:43.652 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:43.310+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:43.653 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:43.507+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:43.653 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:43.507+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:43.653 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:43.507+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:43.653 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:43.507+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:43.916 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:43.886+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:43.916 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:43.886+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:43.916 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:43.886+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:43.916 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:43.886+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:44.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:37:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:37:43.914+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20039 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:37:44.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:44.288+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:44.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:44.288+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:44.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:44.288+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:44.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:44.288+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:44.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:44.471+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:44.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:44.471+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:44.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:44.471+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:44.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:44.472+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:45.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:44.878+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:45.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:44.878+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:45.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:44.878+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:45.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:44.878+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:45.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:45.332+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:45.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:45.332+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:45.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:45.332+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:45.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:45.333+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:45.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:45.472+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:45.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:45.472+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:45.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:45.472+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:45.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:45.473+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:46.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:45.859+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:46.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:45.859+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:46.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:45.859+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:46.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:45.860+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:46.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:46.347+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:46.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:46.347+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:46.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:46.347+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:46.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:46.348+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:46.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:46.436+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:46.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:46.436+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:46.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:46.436+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:46.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:46.437+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:47.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:46.881+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:47.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:46.881+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:47.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:46.881+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:47.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:46.882+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:47.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:47.438+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:47.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:47.438+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:47.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:47.438+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:47.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:47.439+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:47.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:47.336+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:47.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:47.336+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:47.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:47.336+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:47.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:47.337+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:48.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:47.910+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:48.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:47.910+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:48.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:47.910+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:48.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:47.910+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:48.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:48.462+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:48.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:48.462+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:48.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:48.462+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:48.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:48.463+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:48.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:48.375+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:48.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:48.375+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:48.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:48.375+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:48.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:48.375+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:49.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:37:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:37:48.916+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20082 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:37:49.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:48.918+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:49.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:48.918+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:49.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:48.918+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:49.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:48.920+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:49.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:49.385+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:49.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:49.385+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:49.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:49.385+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:49.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:49.386+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:49.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:49.417+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:49.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:49.417+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:49.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:49.417+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:49.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:49.417+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:50.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:49.956+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:50.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:49.956+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:50.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:49.956+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:50.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:49.958+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:50.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:50.429+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:50.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:50.429+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:50.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:50.429+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:50.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:50.429+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:50.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:50.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:50.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:50.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:50.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:50.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:50.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:50.342+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:51.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:50.909+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:51.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:50.909+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:51.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:50.909+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:51.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:50.910+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:51.572 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:51.442+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:51.572 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:51.442+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:51.573 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:51.442+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:51.573 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:51.442+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:51.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:51.311+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:51.574 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:51.311+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:51.574 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:51.311+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:51.574 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:51.312+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:52.358 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:51.932+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:52.358 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:51.932+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:52.358 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:51.932+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:52.358 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:51.932+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:52.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:52.356+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:52.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:52.356+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:52.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:52.356+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:52.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:52.356+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:52.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:52.475+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:52.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:52.475+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:52.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:52.475+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:52.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:52.476+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:53.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:52.889+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:53.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:52.889+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:53.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:52.889+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:53.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:52.890+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:53.652 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:53.430+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:53.652 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:53.430+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:53.653 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:53.430+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:53.653 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:53.430+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:53.653 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:53.363+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:53.653 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:53.363+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:53.653 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:53.363+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:53.653 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:53.364+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:54.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:37:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:37:53.918+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20125 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:37:54.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:53.939+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:54.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:53.939+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:54.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:53.939+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:54.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:53.940+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:54.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:54.349+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:54.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:54.349+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:54.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:54.349+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:54.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:54.349+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:54.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:54.453+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:54.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:54.453+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:54.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:54.453+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:54.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:54.454+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:55.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:54.894+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:55.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:54.894+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:55.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:54.894+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:55.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:54.895+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:55.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:55.417+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:55.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:55.417+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:55.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:55.417+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:55.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:55.418+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:55.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:55.379+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:55.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:55.379+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:55.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:55.379+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:55.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:55.380+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:56.247 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:55.937+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:56.247 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:55.937+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:56.247 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:55.937+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:56.248 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:55.938+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:56.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:56.356+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:56.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:56.356+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:56.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:56.356+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:56.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:56.357+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:56.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:56.433+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:56.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:56.433+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:56.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:56.433+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:56.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:56.433+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:57.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:56.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:57.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:56.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:57.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:56.913+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:57.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:56.914+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:57.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:57.381+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:57.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:57.381+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:57.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:57.381+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:57.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:57.381+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:57.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:57.466+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:57.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:57.466+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:57.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:57.466+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:57.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:57.467+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:58.405 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:57.959+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:58.405 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:57.959+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:58.405 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:57.959+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:58.406 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:57.960+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:58.403+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:58.403+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:58.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:58.403+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:58.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:58.404+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:58.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:58.422+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:58.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:58.422+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:58.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:58.422+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:58.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:58.422+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:59.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:37:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:37:58.920+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20178 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:37:59.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:58.976+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:58.976+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:58.976+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:37:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:58.977+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:37:59.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:59.463+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:59.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:59.463+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:59.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:59.463+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:37:59.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:37:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:37:59.464+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:37:59.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:59.374+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:59.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:59.374+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:59.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:59.374+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:37:59.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:37:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:37:59.374+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:00.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:59.966+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:00.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:59.966+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:00.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:59.966+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:00.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:37:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:37:59.967+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:00.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:00.328+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:00.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:00.328+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:00.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:00.328+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:00.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:00.328+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:00.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:00.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:00.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:00.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:00.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:00.465+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:00.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:00.466+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:01.336 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:00.964+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:01.336 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:00.964+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:01.336 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:00.964+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:01.336 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:00.965+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:01.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:01.334+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:01.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:01.334+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:01.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:01.334+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:01.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:01.335+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:01.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:01.508+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:01.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:01.508+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:01.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:01.508+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:01.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:01.509+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:02.327 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:02.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:02.327 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:02.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:02.328 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:02.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:02.328 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:02.012+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:02.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:02.552+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:02.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:02.552+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:02.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:02.552+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:02.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:02.553+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:02.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:02.325+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:02.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:02.325+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:02.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:02.325+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:02.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:02.326+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:03.345 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:03.004+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:03.346 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:03.004+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:03.346 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:03.004+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:03.346 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:03.005+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:03.596 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:03.593+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:03.596 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:03.343+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:03.596 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:03.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:03.597 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:03.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:03.597 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:03.344+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:03.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:03.593+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:03.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:03.593+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:03.594+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:04.337 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:38:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:38:03.922+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20230 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:38:04.338 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:04.054+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:04.338 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:04.054+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:04.338 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:04.054+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:04.338 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:04.055+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:04.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:04.555+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:04.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:04.555+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:04.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:04.555+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:04.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:04.556+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:04.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:04.335+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:04.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:04.335+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:04.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:04.335+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:04.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:04.336+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:05.375 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:05.029+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:05.375 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:05.029+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:05.376 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:05.029+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:05.376 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:05.030+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:05.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:05.600+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:05.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:05.600+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:05.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:05.600+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:05.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:05.601+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:05.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:05.373+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:05.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:05.373+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:05.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:05.373+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:05.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:05.374+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:06.395 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:06.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:06.396 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:06.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:06.396 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:06.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:06.396 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:06.047+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:06.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:06.609+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:06.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:06.609+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:06.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:06.609+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:06.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:06.609+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:06.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:06.393+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:06.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:06.393+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:06.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:06.393+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:06.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:06.394+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:07.358 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:07.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:07.358 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:07.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:07.358 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:07.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:07.359 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:07.047+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:07.611 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:07.356+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:07.612 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:07.356+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:07.612 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:07.356+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:07.612 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:07.357+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:07.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:07.609+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:07.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:07.609+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:07.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:07.609+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:07.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:07.611+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:08.359 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:08.080+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:08.360 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:08.080+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:08.360 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:08.080+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:08.360 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:08.080+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:08.656 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:08.357+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:08.657 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:08.357+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:08.657 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:08.357+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:08.657 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:08.358+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:08.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:08.655+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:08.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:08.655+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:08.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:08.655+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:08.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:08.656+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:09.346 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:38:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:38:08.924+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20283 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:38:09.346 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:09.108+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:09.347 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:09.108+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:09.347 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:09.108+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:09.347 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:09.110+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:09.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:09.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:09.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:09.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:09.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:09.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:09.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:09.344+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:09.982 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:09.677+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:09.983 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:09.677+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:09.983 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:09.677+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:09.983 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:09.677+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:10.386 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:10.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:10.386 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:10.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:10.386 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:10.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:10.387 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:10.118+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:10.650 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:10.384+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:10.651 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:10.384+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:10.651 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:10.384+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:10.651 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:10.385+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:10.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:10.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:10.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:10.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:10.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:10.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:10.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:10.649+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:11.404 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:11.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:11.404 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:11.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:11.404 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:11.117+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:11.404 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:11.118+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:11.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:11.649+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:11.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:11.649+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:11.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:11.649+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:11.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:11.650+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:11.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:11.402+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:11.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:11.402+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:11.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:11.402+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:11.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:11.402+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:12.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:12.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:12.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:12.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:12.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:12.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:12.421 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:12.131+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:12.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:12.603+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:12.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:12.603+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:12.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:12.603+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:12.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:12.604+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:12.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:12.419+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:12.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:12.419+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:12.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:12.419+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:12.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:12.419+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:13.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:13.084+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:13.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:13.084+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:13.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:13.084+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:13.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:13.084+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:13.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:13.558+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:13.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:13.558+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:13.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:13.558+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:13.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:13.559+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:13.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:13.440+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:13.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:13.440+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:13.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:13.440+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:13.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:13.441+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:14.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:38:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:38:13.926+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20335 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:38:14.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:14.049+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:14.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:14.049+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:14.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:14.049+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:14.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:14.050+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:14.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:14.544+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:14.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:14.544+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:14.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:14.544+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:14.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:14.545+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:14.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:14.453+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:14.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:14.453+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:14.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:14.453+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:14.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:14.454+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:15.415 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:15.001+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:15.415 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:15.001+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:15.415 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:15.001+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:15.416 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:15.001+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:15.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:15.513+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:15.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:15.513+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:15.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:15.513+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:15.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:15.514+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:15.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:15.413+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:15.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:15.413+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:15.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:15.413+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:15.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:15.414+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:16.382 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:16.036+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:16.383 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:16.036+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:16.383 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:16.036+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:16.383 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:16.036+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:16.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:16.380+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:16.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:16.380+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:16.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:16.380+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:16.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:16.381+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:16.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:16.518+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:16.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:16.518+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:16.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:16.518+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:16.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:16.518+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:17.359 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:17.011+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:17.359 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:17.011+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:17.359 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:17.011+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:17.359 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:17.012+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:17.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:17.527+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:17.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:17.527+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:17.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:17.527+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:17.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:17.528+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:17.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:17.356+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:17.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:17.356+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:17.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:17.356+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:17.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:17.357+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:18.334 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:18.009+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:18.334 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:18.009+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:18.334 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:18.009+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:18.334 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:18.010+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:18.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:18.332+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:18.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:18.332+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:18.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:18.332+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:18.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:18.332+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:18.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:18.558+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:18.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:18.558+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:18.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:18.558+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:18.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:18.559+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:19.327 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:38:18.928+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20388 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:38:19.328 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:18.962+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:19.328 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:18.962+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:19.328 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:18.962+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:19.328 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:18.962+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:19.578 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:19.325+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:19.578 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:19.325+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:19.578 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:19.325+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:19.578 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:19.326+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:19.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:19.576+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:19.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:19.576+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:19.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:19.576+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:19.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:19.577+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:20.370 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:19.941+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:20.370 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:19.941+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:20.370 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:19.941+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:20.371 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:19.943+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:20.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:20.368+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:20.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:20.368+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:20.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:20.368+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:20.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:20.369+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:20.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:20.613+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:20.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:20.613+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:20.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:20.613+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:20.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:20.614+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:21.321 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:20.969+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:21.321 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:20.969+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:21.321 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:20.969+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:21.321 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:20.970+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:21.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:21.319+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:21.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:21.319+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:21.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:21.319+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:21.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:21.320+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:21.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:21.611+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:21.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:21.611+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:21.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:21.611+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:21.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:21.612+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:22.274 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:22.016+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:22.275 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:22.016+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:22.275 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:22.016+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:22.275 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:22.017+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:22.574 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:22.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:22.575 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:22.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:22.575 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:22.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:22.575 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:22.273+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:22.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:22.572+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:22.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:22.572+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:22.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:22.572+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:22.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:22.573+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:23.263 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:22.974+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:23.263 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:22.974+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:23.264 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:22.974+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:23.264 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:22.975+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:23.604 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:23.261+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:23.604 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:23.261+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:23.604 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:23.261+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:23.604 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:23.262+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:23.895 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:23.602+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:23.895 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:23.602+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:23.895 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:23.602+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:23.895 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:23.603+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:24.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:38:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:38:23.930+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20440 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:38:24.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:23.931+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:24.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:23.931+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:24.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:23.931+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:24.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:23.933+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:24.649 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:24.275+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:24.649 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:24.275+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:24.649 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:24.275+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:24.649 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:24.275+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:24.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:24.646+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:24.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:24.647+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:24.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:24.647+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:24.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:24.648+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:25.278 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:24.922+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:25.279 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:24.923+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:25.279 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:24.923+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:25.279 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:24.923+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:25.612 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:25.277+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:25.613 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:25.277+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:25.613 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:25.277+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:25.613 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:25.277+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:25.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:25.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:25.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:25.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:25.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:25.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:25.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:25.611+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:26.232 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:25.946+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:26.232 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:25.946+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:26.232 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:25.946+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:26.232 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:25.948+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:26.588 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:26.230+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:26.588 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:26.230+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:26.588 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:26.230+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:26.588 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:26.230+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:26.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:26.586+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:26.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:26.586+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:26.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:26.586+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:26.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:26.587+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:27.221 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:26.965+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:27.221 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:26.965+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:27.222 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:26.965+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:27.222 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:26.966+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:27.625 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:27.219+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:27.625 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:27.219+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:27.625 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:27.219+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:27.625 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:27.220+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:27.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:27.623+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:27.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:27.623+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:27.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:27.623+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:27.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:27.624+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:28.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:28.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:28.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:28.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:28.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:28.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:28.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:28.012+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:28.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:28.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:28.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:28.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:28.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:28.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:28.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:28.199+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:28.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:28.642+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:28.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:28.642+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:28.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:28.642+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:28.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:28.643+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:29.203 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:38:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:38:28.932+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20493 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:38:29.204 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:29.056+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:29.204 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:29.056+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:29.205 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:29.056+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:29.205 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:29.056+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:29.628 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:29.201+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:29.628 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:29.201+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:29.629 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:29.201+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:29.629 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:29.202+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:29.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:29.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:29.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:29.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:29.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:29.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:29.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:29.627+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:30.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:30.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:30.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:30.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:30.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:30.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:30.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:30.177+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:30.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:30.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:30.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:30.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:30.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:30.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:30.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:30.062+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:30.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:30.587+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:30.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:30.587+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:30.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:30.587+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:30.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:30.588+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:31.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:31.214+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:31.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:31.214+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:31.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:31.214+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:31.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:31.215+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:31.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:31.049+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:31.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:31.049+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:31.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:31.049+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:31.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:31.049+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:31.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:31.563+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:31.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:31.563+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:31.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:31.563+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:31.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:31.564+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:32.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:32.044+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:32.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:32.045+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:32.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:32.045+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:32.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:32.045+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:32.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:32.195+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:32.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:32.195+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:32.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:32.195+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:32.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:32.196+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:32.589+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:32.589+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:32.589+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:32.590+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:33.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:33.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:33.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:33.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:33.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:33.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:33.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:33.096+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:33.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:33.189+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:33.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:33.189+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:33.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:33.189+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:33.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:33.190+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:33.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:33.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:33.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:33.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:33.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:33.611+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:34.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:38:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:38:33.934+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20545 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:38:34.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:34.158+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:34.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:34.158+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:34.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:34.158+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:34.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:34.159+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:34.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:34.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:34.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:34.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:34.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:34.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:34.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:34.084+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:34.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:34.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:34.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:34.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:34.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:34.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:34.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:34.595+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:35.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:35.120+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:35.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:35.120+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:35.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:35.120+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:35.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:35.121+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:35.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:35.190+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:35.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:35.190+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:35.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:35.190+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:35.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:35.190+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:35.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:35.577+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:35.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:35.577+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:35.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:35.577+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:35.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:35.578+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:36.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:36.145+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:36.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:36.145+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:36.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:36.145+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:36.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:36.146+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:36.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:36.123+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:36.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:36.123+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:36.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:36.123+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:36.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:36.124+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:36.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:36.590+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:36.590+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:36.590+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:36.591+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:37.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:37.122+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:37.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:37.122+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:37.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:37.122+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:37.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:37.123+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:37.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:37.137+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:37.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:37.137+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:37.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:37.137+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:37.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:37.138+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:37.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:37.585+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:37.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:37.585+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:37.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:37.585+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:37.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:37.586+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:38.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:38.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:38.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:38.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:38.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:38.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:38.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:38.148+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:38.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:38.092+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:38.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:38.092+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:38.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:38.092+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:38.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:38.093+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:38.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:38.606+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:38.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:38.606+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:38.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:38.606+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:38.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:38.607+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:39.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:38:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:38:38.937+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20598 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:38:39.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:39.113+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:39.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:39.113+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:39.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:39.113+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:39.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:39.114+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:39.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:39.095+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:39.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:39.095+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:39.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:39.095+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:39.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:39.096+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:39.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:39.596+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:39.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:39.596+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:39.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:39.596+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:39.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:39.597+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:40.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:40.127+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:40.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:40.127+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:40.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:40.127+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:40.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:40.127+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:40.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:40.108+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:40.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:40.108+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:40.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:40.108+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:40.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:40.109+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:40.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:40.586+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:40.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:40.586+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:40.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:40.586+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:40.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:40.586+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:41.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:41.097+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:41.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:41.097+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:41.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:41.097+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:41.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:41.098+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:41.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:41.114+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:41.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:41.114+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:41.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:41.114+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:41.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:41.114+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:41.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:41.572+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:41.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:41.572+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:41.572+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:41.573+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:42.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:42.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:42.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:42.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:42.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:42.110+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:42.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:42.111+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:42.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:42.152+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:42.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:42.152+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:42.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:42.152+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:42.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:42.153+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:42.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:42.542+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:42.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:42.542+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:42.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:42.542+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:42.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:42.543+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:43.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:43.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:43.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:43.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:43.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:43.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:43.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:43.077+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:43.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:43.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:43.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:43.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:43.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:43.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:43.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:43.142+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:43.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:43.582+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:43.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:43.582+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:43.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:43.582+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:43.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:43.583+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:44.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:38:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:38:43.938+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20650 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:38:44.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:44.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:44.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:44.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:44.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:44.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:44.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:44.148+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:44.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:44.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:44.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:44.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:44.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:44.069+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:44.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:44.069+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:44.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:44.558+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:44.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:44.558+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:44.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:44.558+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:44.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:44.559+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:45.399 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:45.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:45.399 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:45.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:45.399 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:45.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:45.399 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:45.079+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:45.400 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:45.146+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:45.400 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:45.146+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:45.400 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:45.146+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:45.400 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:45.147+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:45.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:45.567+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:45.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:45.567+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:45.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:45.567+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:45.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:45.567+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:46.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:46.110+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:46.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:46.110+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:46.110+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:46.110+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:46.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:46.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:46.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:46.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:46.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:46.063+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:46.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:46.064+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:46.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:46.564+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:46.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:46.564+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:46.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:46.564+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:46.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:46.565+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:47.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:47.034+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:47.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:47.034+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:47.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:47.034+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:47.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:47.034+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:47.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:47.086+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:47.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:47.086+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:47.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:47.086+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:47.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:47.087+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:47.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:47.585+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:47.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:47.585+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:47.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:47.585+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:47.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:47.586+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:48.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:48.078+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:48.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:48.078+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:48.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:48.078+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:48.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:48.079+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:48.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:48.054+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:48.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:48.054+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:48.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:48.054+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:48.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:48.055+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:48.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:48.589+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:48.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:48.589+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:48.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:48.589+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:48.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:48.589+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:49.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:38:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:38:48.940+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20703 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:38:49.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:49.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:49.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:49.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:49.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:49.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:49.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:49.093+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:49.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:49.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:49.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:49.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:49.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:49.070+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:49.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:49.071+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:49.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:49.630+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:49.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:49.630+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:49.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:49.630+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:49.631+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:50.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:50.138+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:50.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:50.138+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:50.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:50.138+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:50.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:50.139+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:50.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:50.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:50.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:50.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:50.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:50.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:50.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:50.077+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:50.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:50.595+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:50.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:50.595+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:50.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:50.595+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:50.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:50.596+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:51.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:51.161+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:51.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:51.161+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:51.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:51.161+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:51.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:51.161+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:51.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:51.042+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:51.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:51.042+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:51.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:51.042+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:51.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:51.043+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:51.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:51.619+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:51.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:51.619+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:51.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:51.619+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:51.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:51.620+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:52.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:52.137+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:52.137+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:52.137+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:52.137+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:52.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:52.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:52.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:52.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:52.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:52.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:52.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:52.059+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:52.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:52.667+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:52.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:52.667+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:52.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:52.667+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:52.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:52.669+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:53.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:53.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:53.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:53.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:53.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:53.129+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:53.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:53.129+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:53.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:53.028+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:53.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:53.028+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:53.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:53.028+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:53.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:53.029+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:53.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:53.641+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:53.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:53.641+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:53.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:53.641+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:53.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:53.642+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:54.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:38:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:38:53.942+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20755 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:38:54.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:54.060+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:54.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:54.060+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:54.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:54.060+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:54.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:54.061+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:54.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:54.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:54.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:54.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:54.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:54.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:54.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:54.084+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:54.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:54.658+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:54.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:54.658+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:54.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:54.658+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:54.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:54.659+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:55.339 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:55.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:55.339 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:55.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:55.339 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:55.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:55.339 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:55.072+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:55.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:55.065+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:55.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:55.065+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:55.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:55.065+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:55.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:55.066+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:55.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:55.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:55.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:55.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:55.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:55.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:55.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:55.654+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:56.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:56.059+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:56.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:56.059+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:56.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:56.059+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:56.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:56.060+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:56.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:56.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:56.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:56.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:56.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:56.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:56.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:56.034+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:56.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:56.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:56.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:56.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:56.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:56.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:56.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:56.610+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:57.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:57.028+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:57.028+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:57.028+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:57.029+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:57.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:57.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:57.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:57.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:57.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:57.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:57.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:57.000+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:57.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:57.595+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:57.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:57.595+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:57.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:57.595+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:57.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:57.596+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:58.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:58.064+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:58.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:58.064+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:58.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:58.064+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:58.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:58.064+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:58.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:57.990+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:58.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:57.990+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:58.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:57.990+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:58.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:57.990+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:58.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:58.623+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:58.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:58.623+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:58.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:58.623+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:58.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:58.624+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:38:59.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:38:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:38:58.944+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20808 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:38:59.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:59.114+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:59.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:59.114+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:59.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:59.114+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:38:59.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:38:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:38:59.115+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:59.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:59.014+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:59.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:59.014+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:59.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:59.014+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:38:59.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:59.015+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:38:59.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:59.639+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:59.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:59.639+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:59.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:59.639+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:38:59.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:38:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:38:59.640+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:00.376 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:00.146+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:00.376 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:00.146+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:00.377 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:00.146+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:00.377 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:00.147+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:00.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:59.993+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:00.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:59.993+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:00.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:59.993+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:00.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:38:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:38:59.993+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:00.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:00.656+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:00.656+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:00.656+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:00.657+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:01.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:01.189+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:01.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:01.189+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:01.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:01.189+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:01.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:01.190+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:01.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:01.038+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:01.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:01.038+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:01.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:01.038+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:01.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:01.038+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:02.061 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:01.680+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:02.061 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:01.680+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:02.061 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:01.680+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:02.061 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:01.681+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:02.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:02.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:02.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:02.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:02.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:02.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:02.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:02.058+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:02.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:02.219+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:02.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:02.219+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:02.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:02.219+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:02.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:02.220+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:02.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:02.636+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:02.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:02.636+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:02.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:02.636+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:02.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:02.637+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:03.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:03.240+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:03.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:03.240+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:03.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:03.240+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:03.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:03.241+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:03.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:03.057+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:03.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:03.057+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:03.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:03.057+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:03.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:03.058+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:03.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:03.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:03.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:03.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:03.654+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:04.242 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:39:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:39:03.946+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20860 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:39:04.243 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:04.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:04.243 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:04.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:04.243 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:04.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:04.243 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:04.023+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:04.630 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:04.239+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:04.630 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:04.239+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:04.631 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:04.239+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:04.631 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:04.240+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:04.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:04.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:04.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:04.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:04.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:04.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:04.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:04.628+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:05.412 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:05.230+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:05.412 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:05.230+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:05.412 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:05.230+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:05.413 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:05.231+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:05.413 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:05.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:05.413 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:05.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:05.413 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:05.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:05.413 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:05.072+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:05.662 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:05.659+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:05.662 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:05.659+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:05.663 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:05.659+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:05.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:05.660+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:06.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:06.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:06.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:06.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:06.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:06.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:06.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:06.034+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:06.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:06.235+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:06.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:06.235+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:06.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:06.235+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:06.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:06.236+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:06.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:06.634+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:06.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:06.634+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:06.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:06.634+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:06.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:06.636+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:07.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:07.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:07.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:07.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:07.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:07.059+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:07.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:07.060+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:07.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:07.189+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:07.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:07.189+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:07.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:07.189+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:07.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:07.190+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:07.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:07.589+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:07.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:07.589+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:07.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:07.589+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:07.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:07.590+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:08.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:08.046+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:08.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:08.046+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:08.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:08.046+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:08.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:08.047+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:08.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:08.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:08.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:08.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:08.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:08.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:08.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:08.189+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:08.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:08.564+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:08.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:08.564+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:08.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:08.564+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:08.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:08.564+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:09.233 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:39:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:39:08.948+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20913 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:39:09.234 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:09.055+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:09.234 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:09.055+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:09.234 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:09.055+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:09.234 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:09.055+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:09.566 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:09.230+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:09.566 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:09.230+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:09.567 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:09.230+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:09.567 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:09.231+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:09.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:09.563+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:09.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:09.563+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:09.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:09.563+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:09.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:09.564+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:10.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:10.051+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:10.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:10.051+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:10.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:10.051+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:10.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:10.051+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:10.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:10.245+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:10.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:10.245+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:10.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:10.245+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:10.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:10.246+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:10.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:10.533+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:10.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:10.533+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:10.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:10.533+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:10.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:10.534+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:11.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:11.252+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:11.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:11.252+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:11.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:11.252+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:11.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:11.252+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:11.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:11.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:11.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:11.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:11.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:11.072+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:11.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:11.073+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:11.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:11.542+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:11.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:11.542+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:11.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:11.542+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:11.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:11.543+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:12.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:12.094+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:12.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:12.094+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:12.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:12.094+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:12.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:12.094+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:12.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:12.281+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:12.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:12.281+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:12.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:12.281+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:12.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:12.281+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:12.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:12.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:12.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:12.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:12.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:12.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:12.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:12.554+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:13.310 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:13.046+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:13.310 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:13.046+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:13.310 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:13.046+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:13.310 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:13.046+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:13.659 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:13.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:13.659 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:13.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:13.660 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:13.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:13.660 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:13.554+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:13.660 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:13.306+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:13.660 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:13.306+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:13.660 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:13.306+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:13.660 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:13.307+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:14.346 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:39:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:39:13.950+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 20965 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:39:14.347 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:14.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:14.347 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:14.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:14.347 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:14.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:14.347 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:14.000+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:14.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:14.554+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:14.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:14.554+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:14.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:14.554+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:14.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:14.555+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:14.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:14.343+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:14.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:14.343+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:14.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:14.343+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:14.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:14.345+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:15.348 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:15.046+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:15.348 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:15.046+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:15.349 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:15.046+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:15.349 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:15.046+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:15.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:15.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:15.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:15.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:15.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:15.346+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:15.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:15.347+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:15.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:15.573+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:15.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:15.573+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:15.677 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:15.573+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:15.677 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:15.574+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:16.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:16.312+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:16.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:16.312+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:16.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:16.312+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:16.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:16.313+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:16.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:16.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:16.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:16.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:16.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:16.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:16.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:16.081+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:16.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:16.527+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:16.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:16.527+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:16.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:16.527+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:16.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:16.528+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:17.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:17.301+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:17.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:17.301+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:17.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:17.301+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:17.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:17.302+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:17.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:17.061+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:17.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:17.061+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:17.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:17.061+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:17.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:17.062+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:17.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:17.549+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:17.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:17.550+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:17.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:17.550+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:17.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:17.550+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:18.344 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:18.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:18.344 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:18.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:18.344 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:18.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:18.344 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:18.090+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:18.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:18.342+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:18.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:18.342+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:18.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:18.342+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:18.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:18.343+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:18.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:18.557+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:18.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:18.557+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:18.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:18.557+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:18.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:18.558+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:19.351 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:39:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:39:18.953+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21018 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:39:19.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:19.114+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:19.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:19.114+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:19.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:19.114+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:19.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:19.115+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:19.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:19.349+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:19.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:19.349+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:19.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:19.349+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:19.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:19.350+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:19.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:19.586+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:19.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:19.586+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:19.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:19.586+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:19.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:19.586+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:20.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:20.350+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:20.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:20.350+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:20.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:20.350+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:20.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:20.351+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:20.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:20.146+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:20.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:20.146+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:20.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:20.146+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:20.428 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:20.147+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:20.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:20.586+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:20.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:20.586+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:20.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:20.586+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:20.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:20.587+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:21.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:21.311+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:21.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:21.311+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:21.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:21.311+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:21.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:21.311+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:21.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:21.124+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:21.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:21.124+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:21.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:21.124+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:21.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:21.125+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:21.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:21.604+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:21.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:21.604+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:21.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:21.604+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:21.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:21.605+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:22.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:22.327+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:22.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:22.327+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:22.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:22.327+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:22.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:22.327+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:22.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:22.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:22.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:22.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:22.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:22.079+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:22.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:22.081+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:22.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:22.618+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:22.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:22.618+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:22.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:22.618+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:22.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:22.619+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:23.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:23.121+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:23.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:23.121+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:23.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:23.121+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:23.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:23.122+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:23.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:23.313+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:23.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:23.313+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:23.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:23.313+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:23.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:23.315+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:23.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:23.593+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:23.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:23.593+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:23.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:23.593+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:23.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:23.595+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:24.282 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:39:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:39:23.955+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21070 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:39:24.282 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:24.161+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:24.282 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:24.161+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:24.283 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:24.161+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:24.283 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:24.161+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:24.594 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:24.280+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:24.594 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:24.280+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:24.594 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:24.280+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:24.594 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:24.281+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:24.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:24.591+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:24.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:24.591+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:24.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:24.591+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:24.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:24.593+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:25.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:25.253+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:25.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:25.253+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:25.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:25.253+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:25.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:25.255+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:25.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:25.154+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:25.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:25.154+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:25.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:25.154+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:25.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:25.155+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:25.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:25.606+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:25.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:25.606+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:25.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:25.606+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:25.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:25.607+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:26.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:26.120+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:26.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:26.120+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:26.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:26.120+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:26.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:26.120+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:26.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:26.290+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:26.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:26.290+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:26.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:26.290+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:26.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:26.291+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:26.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:26.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:26.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:26.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:26.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:26.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:26.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:26.628+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:27.342 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:27.159+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:27.342 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:27.159+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:27.342 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:27.159+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:27.342 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:27.160+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:27.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:27.588+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:27.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:27.588+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:27.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:27.588+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:27.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:27.589+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:27.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:27.339+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:27.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:27.339+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:27.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:27.339+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:27.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:27.341+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:28.424 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:28.141+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:28.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:28.141+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:28.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:28.141+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:28.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:28.141+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:28.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:28.314+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:28.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:28.314+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:28.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:28.314+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:28.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:28.315+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:28.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:28.596+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:28.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:28.596+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:28.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:28.596+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:28.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:28.597+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:29.299 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:39:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:39:28.957+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21123 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:39:29.299 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:29.174+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:29.299 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:29.174+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:29.299 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:29.174+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:29.300 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:29.174+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:29.636 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:29.297+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:29.637 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:29.297+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:29.637 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:29.297+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:29.637 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:29.297+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:29.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:29.634+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:29.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:29.634+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:29.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:29.634+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:29.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:29.635+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:30.596 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:30.306+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:30.596 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:30.306+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:30.596 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:30.306+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:30.596 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:30.307+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:30.597 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:30.188+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:30.597 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:30.188+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:30.597 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:30.188+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:30.597 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:30.189+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:30.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:30.593+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:30.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:30.593+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:30.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:30.593+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:30.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:30.594+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:31.560 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:31.180+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:31.560 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:31.180+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:31.560 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:31.180+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:31.560 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:31.181+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:31.561 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:31.326+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:31.561 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:31.326+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:31.561 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:31.326+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:31.561 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:31.327+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:31.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:31.558+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:31.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:31.558+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:31.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:31.558+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:31.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:31.559+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:32.522 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:32.312+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:32.522 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:32.312+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:32.523 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:32.312+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:32.523 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:32.313+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:32.523 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:32.209+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:32.523 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:32.209+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:32.523 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:32.209+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:32.524 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:32.210+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:32.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:32.520+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:32.520+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:32.520+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:32.521+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:33.524 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:33.255+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:33.525 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:33.255+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:33.525 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:33.255+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:33.525 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:33.256+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:33.525 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:33.323+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:33.525 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:33.323+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:33.525 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:33.323+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:33.526 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:33.323+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:33.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:33.522+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:33.522+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:33.522+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:33.523+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:34.275 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:39:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:39:33.959+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21175 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:39:34.552 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:34.327+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:34.552 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:34.327+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:34.552 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:34.327+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:34.553 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:34.328+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:34.553 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:34.273+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:34.553 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:34.273+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:34.553 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:34.273+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:34.553 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:34.274+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:34.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:34.569+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:34.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:34.569+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:34.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:34.569+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:34.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:34.570+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:35.646 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:35.546+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:35.646 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:35.546+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:35.646 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:35.546+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:35.646 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:35.547+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:35.647 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:35.309+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:35.647 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:35.309+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:35.647 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:35.309+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:35.647 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:35.310+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:35.647 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:35.302+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:35.648 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:35.302+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:35.648 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:35.302+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:35.648 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:35.302+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:36.582 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:36.324+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:36.582 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:36.324+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:36.582 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:36.325+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:36.582 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:36.326+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:36.583 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:36.279+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:36.583 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:36.279+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:36.583 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:36.279+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:36.583 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:36.279+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:36.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:36.580+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:36.580+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:36.580+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:36.581+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:37.601 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:37.322+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:37.601 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:37.322+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:37.601 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:37.322+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:37.601 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:37.323+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:37.601 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:37.334+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:37.602 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:37.334+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:37.602 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:37.334+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:37.602 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:37.335+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:37.602 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:37.533+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:37.603 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:37.533+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:37.603 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:37.533+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:37.603 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:37.534+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:38.489 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:38.323+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:38.489 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:38.323+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:38.489 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:38.323+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:38.489 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:38.324+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:38.490 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:38.299+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:38.490 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:38.299+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:38.490 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:38.299+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:38.490 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:38.299+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:38.896 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:38.572+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:38.896 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:38.572+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:38.896 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:38.572+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:38.897 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:38.572+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:39.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:39:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:39:38.961+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21228 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:39:39.618 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:39.347+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:39.618 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:39.347+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:39.618 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:39.347+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:39.618 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:39.348+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:39.618 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:39.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:39.619 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:39.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:39.619 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:39.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:39.619 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:39.342+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:39.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:39.615+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:39.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:39.615+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:39.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:39.615+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:39.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:39.616+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:40.582 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:40.299+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:40.582 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:40.299+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:40.582 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:40.299+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:40.582 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:40.300+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:40.582 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:40.293+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:40.583 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:40.293+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:40.583 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:40.293+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:40.583 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:40.294+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:40.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:40.580+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:40.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:40.580+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:40.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:40.580+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:40.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:40.580+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:41.549 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:41.264+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:41.549 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:41.264+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:41.549 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:41.264+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:41.550 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:41.264+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:41.550 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:41.300+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:41.550 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:41.300+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:41.550 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:41.300+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:41.550 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:41.301+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:41.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:41.546+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:41.546+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:41.546+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:41.547+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:42.535 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:42.301+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:42.535 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:42.301+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:42.535 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:42.301+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:42.535 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:42.302+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:42.536 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:42.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:42.536 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:42.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:42.536 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:42.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:42.536 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:42.275+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:42.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:42.533+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:42.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:42.533+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:42.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:42.533+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:42.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:42.534+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:43.555 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:43.262+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:43.556 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:43.262+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:43.556 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:43.262+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:43.556 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:43.263+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:43.557 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:43.306+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:43.557 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:43.306+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:43.557 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:43.306+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:43.558 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:43.307+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:43.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:43.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:43.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:43.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:43.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:43.553+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:43.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:43.554+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:44.268 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:39:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:39:43.963+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21280 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:39:44.603 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:44.266+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:44.603 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:44.266+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:44.603 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:44.266+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:44.603 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:44.267+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:44.604 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:44.293+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:44.604 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:44.293+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:44.604 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:44.293+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:44.604 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:44.294+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:44.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:44.601+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:44.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:44.601+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:44.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:44.601+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:44.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:44.602+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:45.590 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:45.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:45.590 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:45.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:45.590 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:45.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:45.590 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:45.295+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:45.591 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:45.242+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:45.591 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:45.242+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:45.591 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:45.242+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:45.591 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:45.244+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:45.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:45.588+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:45.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:45.588+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:45.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:45.588+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:45.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:45.588+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:46.627 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:46.240+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:46.628 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:46.240+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:46.628 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:46.240+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:46.628 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:46.241+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:46.628 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:46.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:46.629 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:46.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:46.629 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:46.340+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:46.629 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:46.340+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:46.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:46.625+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:46.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:46.625+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:46.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:46.625+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:46.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:46.626+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:47.597 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:47.279+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:47.597 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:47.279+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:47.597 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:47.279+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:47.597 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:47.279+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:47.597 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:47.331+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:47.598 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:47.331+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:47.598 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:47.331+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:47.598 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:47.332+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:47.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:47.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:47.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:47.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:47.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:47.594+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:47.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:47.595+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:48.629 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:48.379+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:48.629 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:48.379+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:48.629 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:48.379+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:48.629 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:48.379+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:48.629 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:48.284+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:48.630 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:48.284+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:48.630 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:48.284+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:48.630 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:48.285+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:48.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:48.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:48.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:48.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:48.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:48.626+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:48.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:48.628+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:49.264 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:39:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:39:48.965+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21333 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:39:49.562 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:49.375+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:49.562 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:49.375+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:49.562 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:49.375+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:49.562 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:49.376+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:49.563 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:49.262+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:49.563 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:49.262+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:49.563 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:49.262+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:49.563 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:49.264+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:49.619+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:49.619+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:49.619+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:49.620+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:50.613 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:50.292+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:50.613 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:50.292+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:50.613 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:50.292+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:50.614 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:50.292+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:50.614 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:50.349+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:50.614 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:50.349+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:50.614 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:50.349+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:50.614 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:50.350+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:50.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:50.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:50.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:50.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:50.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:50.610+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:50.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:50.611+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:51.570 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:51.269+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:51.570 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:51.269+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:51.570 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:51.269+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:51.570 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:51.270+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:51.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:51.373+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:51.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:51.373+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:51.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:51.373+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:51.571 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:51.374+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:51.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:51.566+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:51.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:51.566+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:51.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:51.566+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:51.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:51.567+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:52.613 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:52.359+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:52.614 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:52.359+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:52.614 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:52.359+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:52.614 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:52.359+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:52.614 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:52.300+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:52.614 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:52.300+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:52.615 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:52.300+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:52.615 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:52.301+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:52.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:52.611+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:52.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:52.611+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:52.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:52.611+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:52.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:52.612+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:53.662 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:53.308+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:53.662 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:53.308+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:53.662 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:53.308+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:53.662 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:53.309+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:53.662 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:53.332+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:53.663 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:53.332+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:53.663 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:53.332+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:53.663 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:53.332+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:53.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:53.660+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:53.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:53.660+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:53.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:53.660+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:53.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:53.661+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:54.305 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:39:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:39:53.967+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21385 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:39:54.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:54.303+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:54.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:54.303+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:54.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:54.303+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:54.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:54.304+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:54.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:54.369+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:54.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:54.369+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:54.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:54.369+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:54.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:54.370+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:55.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:54.696+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:55.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:54.696+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:55.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:54.696+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:55.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:54.697+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:55.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:55.332+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:55.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:55.332+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:55.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:55.332+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:55.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:55.333+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:55.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:55.348+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:55.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:55.348+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:55.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:55.348+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:55.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:55.349+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:56.055 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:55.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:56.055 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:55.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:56.055 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:55.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:56.056 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:55.679+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:56.381 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:56.290+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:56.381 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:56.290+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:56.382 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:56.290+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:56.382 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:56.291+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:56.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:56.379+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:56.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:56.379+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:56.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:56.379+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:56.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:56.380+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:57.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:56.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:57.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:56.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:57.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:56.706+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:57.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:56.707+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:57.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:57.390+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:57.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:57.390+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:57.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:57.390+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:57.671 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:57.391+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:57.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:57.328+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:57.671 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:57.328+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:57.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:57.328+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:57.672 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:57.328+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:57.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:57.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:57.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:57.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:57.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:57.668+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:57.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:57.669+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:58.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:58.438+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:58.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:58.438+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:58.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:58.438+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:58.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:58.439+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:58.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:58.309+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:58.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:58.309+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:58.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:58.309+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:58.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:58.310+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:58.972 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:58.680+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:58.972 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:58.680+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:58.972 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:58.680+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:58.972 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:58.681+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:39:59.315 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:39:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:39:58.969+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21438 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:39:59.665 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:59.444+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:59.665 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:59.444+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:59.665 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:59.444+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:39:59.665 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:39:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:39:59.445+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:59.665 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:59.313+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:59.666 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:59.313+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:59.666 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:59.313+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:39:59.666 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:39:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:39:59.313+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:39:59.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:59.663+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:59.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:59.663+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:59.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:59.663+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:39:59.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:39:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:39:59.663+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:00.642 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:00.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:00.643 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:00.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:00.643 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:00.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:00.643 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:00.489+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:00.644 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:00.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:00.644 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:00.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:00.644 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:00.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:00.644 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:00.311+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:00.640+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:00.640+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:00.640+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:00.641+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:01.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:01.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:01.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:01.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:01.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:01.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:01.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:01.461+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:01.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:01.279+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:01.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:01.279+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:01.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:01.279+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:01.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:01.280+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:01.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:01.655+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:01.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:01.655+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:01.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:01.655+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:01.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:01.657+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:02.658 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:02.305+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:02.658 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:02.305+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:02.659 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:02.305+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:02.659 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:02.306+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:02.659 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:02.440+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:02.659 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:02.440+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:02.659 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:02.440+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:02.659 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:02.441+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:02.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:02.656+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:02.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:02.656+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:02.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:02.656+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:02.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:02.657+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:03.644 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:03.398+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:03.645 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:03.398+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:03.645 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:03.398+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:03.645 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:03.399+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:03.645 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:03.335+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:03.645 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:03.335+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:03.646 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:03.335+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:03.646 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:03.335+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:03.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:03.642+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:03.642+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:03.642+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:03.643+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:04.352 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:40:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:40:03.971+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21490 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:40:04.641 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:04.421+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:04.641 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:04.421+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:04.641 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:04.421+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:04.642 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:04.421+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:04.642 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:04.350+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:04.642 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:04.350+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:04.642 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:04.350+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:04.642 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:04.351+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:04.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:04.639+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:04.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:04.639+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:04.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:04.639+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:04.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:04.640+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:05.656 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:05.452+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:05.657 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:05.452+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:05.657 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:05.452+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:05.657 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:05.453+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:05.657 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:05.383+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:05.657 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:05.383+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:05.657 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:05.383+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:05.658 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:05.383+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:05.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:05.654+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:05.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:05.654+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:05.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:05.654+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:05.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:05.655+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:06.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:06.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:06.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:06.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:06.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:06.489+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:06.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:06.490+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:06.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:06.403+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:06.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:06.403+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:06.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:06.403+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:06.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:06.404+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:07.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:06.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:07.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:06.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:07.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:06.697+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:07.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:06.698+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:07.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:07.380+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:07.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:07.380+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:07.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:07.380+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:07.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:07.380+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:07.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:07.539+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:07.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:07.539+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:07.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:07.539+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:07.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:07.540+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:08.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:07.728+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:08.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:07.728+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:08.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:07.728+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:08.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:07.729+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:08.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:08.543+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:08.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:08.543+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:08.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:08.543+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:08.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:08.544+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:08.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:08.371+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:08.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:08.371+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:08.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:08.371+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:08.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:08.372+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:09.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:40:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:40:08.974+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21543 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:40:09.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:08.742+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:09.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:08.742+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:09.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:08.742+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:09.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:08.743+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:09.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:09.529+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:09.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:09.529+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:09.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:09.529+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:09.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:09.529+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:09.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:09.421+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:09.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:09.421+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:09.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:09.421+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:09.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:09.422+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:10.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:09.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:10.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:09.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:10.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:09.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:10.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:09.749+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:10.706 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:10.552+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:10.706 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:10.552+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:10.706 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:10.552+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:10.706 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:10.553+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:10.706 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:10.443+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:10.707 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:10.443+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:10.707 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:10.443+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:10.707 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:10.444+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:11.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:10.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:11.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:10.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:11.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:10.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:11.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:10.704+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:11.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:11.540+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:11.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:11.540+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:11.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:11.540+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:11.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:11.541+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:11.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:11.405+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:11.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:11.405+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:11.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:11.405+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:11.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:11.406+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:12.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:11.754+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:12.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:11.754+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:12.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:11.754+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:12.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:11.755+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:12.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:12.361+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:12.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:12.361+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:12.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:12.361+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:12.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:12.362+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:12.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:12.492+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:12.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:12.492+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:12.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:12.492+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:12.677 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:12.494+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:13.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:12.767+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:13.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:12.767+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:13.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:12.767+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:13.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:12.768+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:13.664 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:13.520+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:13.664 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:13.520+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:13.664 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:13.520+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:13.664 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:13.521+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:13.664 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:13.406+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:13.665 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:13.406+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:13.665 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:13.406+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:13.665 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:13.407+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:13.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:13.762+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:13.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:13.762+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:13.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:13.762+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:13.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:13.763+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:14.385 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:40:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:40:13.975+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21595 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:40:14.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:14.535+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:14.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:14.535+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:14.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:14.535+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:14.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:14.536+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:14.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:14.382+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:14.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:14.382+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:14.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:14.382+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:14.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:14.383+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:15.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:14.804+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:15.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:14.804+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:15.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:14.804+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:15.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:14.806+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:15.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:15.386+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:15.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:15.386+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:15.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:15.386+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:15.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:15.386+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:15.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:15.540+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:15.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:15.540+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:15.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:15.540+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:15.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:15.541+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:16.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:15.825+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:16.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:15.825+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:16.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:15.825+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:16.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:15.826+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:16.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:16.366+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:16.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:16.366+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:16.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:16.366+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:16.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:16.367+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:16.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:16.549+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:16.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:16.549+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:16.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:16.549+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:16.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:16.550+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:17.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:16.792+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:17.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:16.792+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:17.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:16.792+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:17.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:16.792+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:17.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:17.334+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:17.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:17.334+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:17.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:17.334+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:17.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:17.335+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:17.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:17.518+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:17.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:17.518+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:17.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:17.518+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:17.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:17.520+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:18.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:17.821+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:18.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:17.821+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:18.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:17.821+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:18.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:17.822+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:18.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:18.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:18.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:18.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:18.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:18.342+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:18.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:18.342+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:18.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:18.565+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:18.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:18.565+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:18.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:18.565+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:18.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:18.566+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:19.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:40:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:40:18.977+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21648 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:40:19.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:18.865+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:19.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:18.865+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:19.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:18.865+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:19.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:18.866+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:19.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:19.370+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:19.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:19.370+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:19.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:19.370+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:19.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:19.370+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:19.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:19.600+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:19.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:19.600+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:19.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:19.600+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:19.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:19.601+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:20.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:19.907+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:20.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:19.907+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:20.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:19.907+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:20.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:19.908+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:20.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:20.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:20.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:20.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:20.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:20.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:20.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:20.408+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:20.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:20.629+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:20.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:20.629+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:20.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:20.629+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:20.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:20.630+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:21.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:20.911+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:21.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:20.911+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:21.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:20.911+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:21.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:20.912+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:21.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:21.412+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:21.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:21.412+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:21.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:21.412+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:21.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:21.413+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:21.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:21.655+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:21.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:21.655+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:21.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:21.655+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:21.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:21.656+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:22.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:21.924+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:22.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:21.924+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:22.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:21.924+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:22.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:21.925+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:22.708 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:22.453+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:22.708 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:22.453+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:22.708 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:22.453+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:22.708 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:22.454+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:23.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:22.705+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:23.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:22.705+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:23.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:22.705+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:23.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:22.706+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:23.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:22.921+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:23.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:22.921+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:23.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:22.921+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:23.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:22.922+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:23.751 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:23.494+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:23.751 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:23.494+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:23.752 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:23.494+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:23.752 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:23.495+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:24.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:40:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:40:23.980+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21700 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:40:24.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:23.749+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:24.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:23.749+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:24.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:23.749+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:24.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:23.750+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:24.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:23.962+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:24.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:23.962+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:24.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:23.962+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:24.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:23.964+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:24.742 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:24.460+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:24.743 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:24.460+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:24.743 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:24.460+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:24.743 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:24.461+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:25.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:24.736+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:25.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:24.736+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:25.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:24.736+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:25.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:24.737+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:25.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:24.984+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:25.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:24.984+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:25.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:24.984+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:25.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:24.985+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:25.758 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:25.465+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:25.758 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:25.465+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:25.759 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:25.465+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:25.759 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:25.465+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:26.058 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:25.756+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:26.059 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:25.756+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:26.059 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:25.756+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:26.059 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:25.757+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:26.059 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:26.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:26.059 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:26.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:26.060 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:26.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:26.060 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:26.000+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:26.754 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:26.456+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:26.754 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:26.456+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:26.755 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:26.456+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:26.755 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:26.457+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:27.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:26.752+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:27.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:26.752+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:27.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:26.752+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:27.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:26.753+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:27.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:26.992+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:27.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:26.992+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:27.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:26.992+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:27.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:26.992+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:27.782 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:27.467+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:27.782 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:27.467+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:27.783 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:27.467+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:27.783 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:27.468+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:28.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:27.780+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:28.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:27.780+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:28.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:27.780+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:28.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:27.781+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:28.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:28.015+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:28.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:28.015+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:28.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:28.015+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:28.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:28.015+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:28.822 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:28.439+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:28.823 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:28.439+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:28.823 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:28.440+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:28.823 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:28.440+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:29.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:40:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:40:28.982+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21753 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:40:29.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:29.013+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:29.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:29.013+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:29.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:29.013+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:29.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:29.013+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:29.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:28.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:29.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:28.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:29.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:28.820+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:29.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:28.821+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:29.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:29.405+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:29.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:29.405+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:29.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:29.405+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:29.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:29.406+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:30.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:29.870+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:30.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:29.870+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:30.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:29.870+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:30.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:29.871+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:30.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:30.011+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:30.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:30.012+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:30.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:30.012+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:30.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:30.013+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:30.909 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:30.438+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:30.910 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:30.438+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:30.910 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:30.438+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:30.910 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:30.439+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:31.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:30.907+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:31.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:30.907+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:31.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:30.907+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:31.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:30.908+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:31.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:31.052+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:31.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:31.052+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:31.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:31.052+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:31.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:31.053+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:31.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:31.463+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:31.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:31.463+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:31.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:31.463+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:31.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:31.465+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:32.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:31.927+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:32.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:31.927+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:32.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:31.927+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:32.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:31.928+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:32.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:32.079+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:32.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:32.079+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:32.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:32.079+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:32.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:32.079+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:32.916 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:32.473+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:32.916 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:32.473+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:32.917 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:32.473+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:32.917 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:32.473+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:33.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:32.914+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:33.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:32.914+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:33.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:32.914+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:33.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:32.915+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:33.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:33.030+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:33.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:33.030+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:33.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:33.030+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:33.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:33.032+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:33.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:33.523+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:33.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:33.523+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:33.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:33.523+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:33.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:33.524+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:34.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:40:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:40:33.984+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21805 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:40:34.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:33.924+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:34.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:33.924+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:34.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:33.924+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:34.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:33.925+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:34.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:34.044+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:34.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:34.044+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:34.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:34.044+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:34.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:34.046+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:34.895 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:34.490+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:34.895 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:34.490+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:34.895 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:34.490+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:34.896 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:34.491+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:35.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:34.892+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:35.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:34.892+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:35.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:34.892+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:35.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:34.893+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:35.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:35.026+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:35.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:35.026+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:35.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:35.026+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:35.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:35.027+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:35.888 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:35.505+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:35.889 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:35.505+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:35.889 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:35.505+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:35.889 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:35.506+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:36.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:35.885+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:36.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:35.885+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:36.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:35.885+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:36.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:35.886+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:36.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:36.044+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:36.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:36.044+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:36.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:36.044+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:36.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:36.044+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:36.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:36.546+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:36.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:36.546+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:36.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:36.546+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:36.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:36.546+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:37.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:36.930+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:37.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:36.930+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:37.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:36.930+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:37.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:36.931+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:37.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:37.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:37.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:37.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:37.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:37.005+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:37.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:37.006+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:37.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:37.535+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:37.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:37.536+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:37.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:37.536+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:37.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:37.536+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:38.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:37.937+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:38.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:37.937+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:38.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:37.937+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:38.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:37.938+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:38.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:37.975+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:38.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:37.975+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:38.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:37.975+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:38.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:37.976+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:38.906 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:38.519+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:38.907 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:38.519+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:38.907 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:38.519+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:38.907 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:38.520+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:39.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:40:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:40:38.985+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21858 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:40:39.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:38.903+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:39.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:38.903+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:39.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:38.903+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:39.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:38.904+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:39.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:38.991+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:39.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:38.991+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:39.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:38.991+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:39.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:38.992+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:39.882 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:39.521+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:39.882 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:39.521+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:39.882 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:39.521+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:39.882 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:39.522+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:40.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:39.956+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:40.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:39.956+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:40.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:39.956+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:40.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:39.956+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:40.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:39.879+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:40.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:39.879+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:40.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:39.879+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:40.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:39.880+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:40.907 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:40.568+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:40.908 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:40.568+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:40.908 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:40.568+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:40.908 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:40.569+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:41.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:40.904+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:41.172 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:40.904+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:41.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:40.904+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:41.173 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:40.905+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:41.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:40.967+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:41.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:40.967+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:41.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:40.967+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:41.173 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:40.968+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:41.918 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:41.579+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:41.919 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:41.579+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:41.919 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:41.579+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:41.919 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:41.580+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:42.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:41.915+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:42.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:41.915+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:42.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:41.915+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:42.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:41.916+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:42.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:41.942+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:42.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:41.942+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:42.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:41.942+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:42.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:41.943+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:42.882 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:42.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:42.883 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:42.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:42.883 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:42.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:42.883 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:42.561+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:43.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:42.913+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:43.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:42.913+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:43.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:42.913+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:43.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:42.914+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:43.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:42.880+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:43.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:42.880+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:43.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:42.880+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:43.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:42.880+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:43.889 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:43.526+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:43.890 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:43.526+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:43.890 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:43.526+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:43.890 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:43.527+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:44.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:40:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:40:43.987+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21910 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:40:44.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:43.936+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:44.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:43.936+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:44.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:43.936+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:44.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:43.937+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:44.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:43.886+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:44.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:43.886+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:44.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:43.886+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:44.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:43.888+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:44.902 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:44.560+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:44.902 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:44.560+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:44.902 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:44.560+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:44.902 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:44.561+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:45.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:44.930+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:45.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:44.930+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:45.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:44.930+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:45.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:44.931+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:45.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:44.898+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:45.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:44.898+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:45.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:44.898+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:45.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:44.899+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:45.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:45.564+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:45.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:45.564+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:45.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:45.564+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:45.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:45.564+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:46.211 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:45.936+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:46.212 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:45.936+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:46.212 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:45.936+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:46.212 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:45.937+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:46.212 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:45.945+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:46.212 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:45.945+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:46.212 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:45.945+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:46.213 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:45.946+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:46.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:46.590+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:46.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:46.590+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:46.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:46.590+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:46.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:46.591+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:47.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:46.922+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:47.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:46.922+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:47.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:46.922+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:47.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:46.922+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:47.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:46.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:47.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:46.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:47.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:46.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:47.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:46.973+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:47.905 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:47.602+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:47.905 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:47.602+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:47.905 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:47.602+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:47.905 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:47.602+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:48.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:47.901+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:48.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:47.901+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:48.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:47.901+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:48.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:47.902+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:48.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:48.009+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:48.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:48.009+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:48.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:48.009+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:48.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:48.010+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:48.892 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:48.560+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:48.893 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:48.560+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:48.893 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:48.560+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:48.893 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:48.561+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:49.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:40:48.990+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 21963 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:40:49.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:48.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:49.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:48.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:49.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:48.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:49.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:48.980+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:49.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:48.890+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:49.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:48.890+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:49.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:48.890+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:49.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:48.891+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:49.913 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:49.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:49.913 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:49.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:49.913 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:49.561+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:49.913 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:49.561+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:50.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:49.941+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:50.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:49.941+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:50.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:49.941+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:50.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:49.942+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:50.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:49.911+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:50.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:49.911+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:50.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:49.911+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:50.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:49.912+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:50.901 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:50.565+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:50.902 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:50.565+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:50.902 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:50.565+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:50.902 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:50.565+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:51.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:50.932+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:51.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:50.932+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:51.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:50.932+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:51.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:50.932+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:51.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:50.899+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:51.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:50.899+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:51.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:50.899+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:51.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:50.900+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:51.857 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:51.604+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:51.857 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:51.604+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:51.857 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:51.604+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:51.857 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:51.605+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:52.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:51.908+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:52.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:51.908+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:52.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:51.908+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:52.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:51.909+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:52.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:51.854+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:52.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:51.854+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:52.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:51.854+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:52.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:51.855+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:52.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:52.876+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:52.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:52.876+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:52.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:52.876+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:52.926 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:52.877+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:52.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:52.825+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:52.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:52.825+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:52.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:52.825+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:52.927 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:52.825+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:52.927 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:52.652+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:52.927 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:52.652+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:52.927 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:52.652+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:52.928 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:52.652+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:53.891 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:53.783+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:53.891 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:53.783+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:53.891 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:53.783+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:53.891 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:53.784+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:53.892 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:53.623+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:53.892 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:53.623+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:53.892 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:53.623+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:53.892 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:53.623+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:54.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:40:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:40:53.992+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22015 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:40:54.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:53.889+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:54.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:53.889+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:54.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:53.889+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:54.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:53.889+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:54.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:54.654+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:54.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:54.654+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:54.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:54.654+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:54.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:54.655+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:55.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:54.841+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:55.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:54.841+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:55.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:54.841+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:55.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:54.842+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:55.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:54.798+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:55.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:54.798+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:55.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:54.798+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:55.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:54.799+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:55.877 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:55.829+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:55.877 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:55.829+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:55.877 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:55.829+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:55.878 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:55.829+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:55.878 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:55.609+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:55.878 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:55.609+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:55.878 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:55.609+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:55.878 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:55.609+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:56.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:55.875+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:56.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:55.875+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:56.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:55.875+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:56.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:55.875+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:56.881 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:56.617+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:56.881 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:56.617+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:56.881 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:56.617+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:56.881 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:56.618+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:57.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:56.905+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:57.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:56.905+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:57.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:56.905+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:57.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:56.905+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:57.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:56.878+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:57.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:56.878+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:57.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:56.878+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:57.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:56.879+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:57.860 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:57.586+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:57.860 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:57.586+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:57.860 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:57.586+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:57.860 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:57.587+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:58.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:57.935+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:58.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:57.935+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:58.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:57.935+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:58.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:57.936+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:58.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:57.858+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:58.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:57.858+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:58.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:57.858+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:58.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:57.859+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:58.880 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:58.595+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:58.880 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:58.595+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:58.881 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:58.595+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:58.881 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:58.596+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:59.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:40:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:40:58.994+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22068 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:40:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:58.949+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:58.949+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:58.949+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:40:59.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:58.950+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:40:59.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:58.878+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:59.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:58.878+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:59.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:58.878+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:59.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:58.879+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:59.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:59.846+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:59.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:59.846+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:59.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:59.846+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:40:59.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:40:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:40:59.847+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:40:59.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:59.638+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:59.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:59.638+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:59.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:59.638+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:40:59.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:40:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:40:59.639+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:00.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:59.945+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:00.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:59.945+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:00.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:59.945+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:00.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:40:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:40:59.946+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:00.914 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:00.641+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:00.915 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:00.641+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:00.915 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:00.641+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:00.915 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:00.642+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:00.915 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:00.863+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:00.915 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:00.863+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:00.915 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:00.863+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:00.916 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:00.865+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:01.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:00.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:01.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:00.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:01.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:00.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:01.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:00.913+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:01.908 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:01.876+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:01.908 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:01.876+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:01.908 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:01.876+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:01.908 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:01.878+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:01.909 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:01.638+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:01.909 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:01.638+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:01.909 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:01.638+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:01.909 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:01.639+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:02.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:01.905+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:02.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:01.905+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:02.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:01.905+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:02.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:01.906+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:02.883 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:02.840+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:02.883 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:02.840+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:02.883 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:02.840+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:02.884 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:02.841+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:02.884 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:02.600+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:02.884 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:02.600+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:02.884 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:02.600+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:02.884 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:02.601+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:03.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:02.881+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:03.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:02.881+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:03.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:02.881+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:03.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:02.881+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:03.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:03.639+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:03.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:03.639+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:03.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:03.639+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:03.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:03.640+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:03.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:03.880+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:03.926 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:03.880+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:03.926 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:03.880+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:03.926 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:03.881+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:03.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:03.853+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:03.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:03.853+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:03.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:03.853+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:03.927 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:03.854+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:04.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:41:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:41:03.997+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22118 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:41:05.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:04.911+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:05.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:04.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:05.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:04.912+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:05.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:04.912+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:05.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:04.688+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:05.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:04.688+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:05.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:04.688+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:05.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:04.688+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:05.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:04.841+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:05.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:04.841+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:05.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:04.841+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:05.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:04.842+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:05.917 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:05.829+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:05.917 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:05.829+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:05.918 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:05.829+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:05.918 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:05.830+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:05.918 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:05.661+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:05.918 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:05.661+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:05.918 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:05.661+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:05.918 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:05.661+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:06.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:05.915+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:06.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:05.915+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:06.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:05.915+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:06.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:05.916+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:06.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:06.636+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:06.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:06.636+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:06.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:06.636+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:06.922 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:06.637+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:06.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:06.790+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:06.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:06.790+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:06.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:06.790+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:06.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:06.791+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:07.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:06.920+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:07.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:06.920+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:07.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:06.920+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:07.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:06.921+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:08.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:07.763+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:08.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:07.763+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:08.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:07.763+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:08.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:07.764+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:08.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:07.916+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:08.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:07.916+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:08.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:07.916+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:08.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:07.918+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:08.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:07.684+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:08.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:07.684+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:08.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:07.684+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:08.177 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:07.684+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:09.001 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:08.941+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:09.002 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:08.941+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:09.002 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:08.941+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:09.002 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:08.942+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:09.002 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:08.707+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:09.003 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:08.707+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:09.003 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:08.707+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:09.003 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:08.707+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:09.003 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:08.768+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:09.003 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:08.768+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:09.004 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:08.768+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:09.004 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:08.769+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:09.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:41:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:41:08.999+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22171 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:41:09.945 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:09.789+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:09.946 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:09.789+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:09.946 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:09.789+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:09.946 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:09.789+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:09.946 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:09.688+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:09.946 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:09.688+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:09.946 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:09.688+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:09.947 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:09.688+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:10.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:09.943+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:10.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:09.943+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:10.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:09.943+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:10.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:09.944+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:10.995 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:10.761+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:10.995 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:10.761+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:10.995 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:10.761+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:10.995 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:10.762+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:10.996 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:10.680+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:10.996 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:10.680+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:10.996 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:10.680+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:10.996 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:10.681+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:11.407 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:10.993+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:11.407 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:10.993+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:11.407 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:10.993+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:11.407 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:10.994+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:11.982 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:11.722+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:11.982 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:11.722+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:11.982 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:11.722+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:11.983 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:11.722+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:11.983 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:11.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:11.983 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:11.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:11.983 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:11.731+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:11.983 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:11.732+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:12.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:11.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:12.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:11.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:12.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:11.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:12.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:11.980+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:13.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:12.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:13.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:12.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:13.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:12.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:13.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:12.976+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:13.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:12.738+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:13.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:12.738+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:13.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:12.738+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:13.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:12.738+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:13.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:12.738+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:13.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:12.738+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:13.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:12.738+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:13.177 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:12.739+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:13.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:13.743+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:13.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:13.743+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:13.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:13.743+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:13.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:13.744+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:13.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:13.784+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:13.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:13.784+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:13.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:13.784+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:13.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:13.786+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:14.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:41:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:41:14.001+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22223 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:41:14.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:13.938+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:14.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:13.938+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:14.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:13.938+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:14.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:13.939+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:15.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:14.916+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:15.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:14.916+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:15.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:14.916+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:15.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:14.917+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:15.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:14.786+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:15.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:14.786+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:15.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:14.786+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:15.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:14.787+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:15.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:14.757+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:15.177 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:14.757+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:15.177 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:14.757+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:15.177 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:14.758+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:16.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:15.938+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:16.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:15.938+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:16.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:15.938+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:16.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:15.939+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:16.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:15.807+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:16.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:15.807+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:16.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:15.807+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:16.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:15.808+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:16.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:15.737+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:16.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:15.737+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:16.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:15.737+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:16.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:15.738+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:17.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:16.699+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:17.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:16.699+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:17.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:16.699+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:17.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:16.700+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:17.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:16.936+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:17.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:16.936+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:17.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:16.936+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:17.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:16.937+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:17.177 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:16.799+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:17.177 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:16.799+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:17.177 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:16.799+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:17.178 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:16.800+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:17.981 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:17.785+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:17.981 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:17.785+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:17.981 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:17.785+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:17.981 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:17.787+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:17.982 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:17.700+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:17.982 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:17.700+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:17.982 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:17.700+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:17.982 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:17.701+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:18.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:17.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:18.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:17.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:18.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:17.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:18.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:17.979+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:18.968 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:18.742+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:18.968 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:18.742+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:18.968 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:18.742+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:18.968 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:18.743+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:18.969 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:18.965+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:18.969 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:18.965+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:18.969 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:18.965+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:18.969 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:18.966+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:18.970 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:18.716+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:18.970 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:18.716+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:18.970 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:18.716+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:18.970 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:18.716+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:19.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:41:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:41:19.004+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22275 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:41:20.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:19.763+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:20.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:19.763+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:20.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:19.763+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:20.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:19.764+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:20.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:19.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:20.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:19.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:20.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:19.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:20.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:19.979+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:20.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:19.742+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:20.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:19.742+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:20.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:19.742+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:20.177 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:19.742+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:21.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:20.783+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:21.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:20.783+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:21.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:20.783+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:21.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:20.783+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:21.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:20.993+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:21.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:20.993+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:21.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:20.993+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:21.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:20.993+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:21.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:20.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:21.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:20.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:21.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:20.746+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:21.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:20.747+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:21.768 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:21.730+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:21.769 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:21.730+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:21.769 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:21.730+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:21.769 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:21.731+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:22.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:21.953+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:22.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:21.953+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:22.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:21.953+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:22.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:21.955+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:22.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:21.766+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:22.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:21.766+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:22.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:21.766+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:22.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:21.767+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:23.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:22.906+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:23.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:22.906+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:23.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:22.906+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:23.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:22.907+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:23.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:22.771+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:23.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:22.771+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:23.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:22.771+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:23.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:22.772+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:23.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:22.801+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:23.177 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:22.801+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:23.177 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:22.801+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:23.177 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:22.802+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:23.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:23.777+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:23.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:23.777+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:23.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:23.777+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:23.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:23.778+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:23.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:23.786+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:23.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:23.786+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:23.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:23.786+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:23.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:23.787+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:24.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:41:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:41:24.006+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22327 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:41:24.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:23.931+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:24.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:23.931+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:24.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:23.931+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:24.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:23.932+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:25.143 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:24.942+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:25.143 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:24.942+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:25.144 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:24.942+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:25.144 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:24.943+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:25.144 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:24.835+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:25.144 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:24.835+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:25.144 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:24.835+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:25.145 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:24.835+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:25.145 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:24.783+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:25.145 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:24.783+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:25.145 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:24.783+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:25.145 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:24.784+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:26.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:25.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:26.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:25.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:26.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:25.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:26.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:25.956+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:26.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:25.858+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:26.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:25.858+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:26.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:25.858+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:26.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:25.859+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:26.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:25.823+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:26.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:25.823+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:26.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:25.823+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:26.177 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:25.824+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:27.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:26.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:27.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:26.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:27.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:26.995+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:27.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:26.996+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:27.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:26.878+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:27.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:26.878+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:27.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:26.878+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:27.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:26.879+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:27.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:26.776+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:27.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:26.776+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:27.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:26.776+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:27.177 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:26.777+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:28.174 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:28.038+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:28.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:28.038+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:28.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:28.038+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:28.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:28.038+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:28.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:27.834+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:28.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:27.834+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:28.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:27.834+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:28.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:27.835+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:28.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:27.814+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:28.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:27.814+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:28.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:27.814+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:28.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:27.815+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:29.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:41:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:41:29.008+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22380 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:41:29.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:29.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:29.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:29.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:29.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:29.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:29.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:29.022+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:29.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:28.797+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:29.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:28.797+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:29.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:28.797+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:29.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:28.798+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:29.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:28.803+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:29.177 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:28.803+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:29.177 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:28.803+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:29.177 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:28.803+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:30.073 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:29.818+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:30.073 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:29.818+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:30.073 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:29.818+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:30.073 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:29.819+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:30.073 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:29.815+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:30.074 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:29.815+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:30.074 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:29.815+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:30.074 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:29.815+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:30.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:30.070+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:30.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:30.070+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:30.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:30.070+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:30.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:30.071+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:31.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:30.819+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:31.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:30.819+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:31.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:30.819+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:31.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:30.820+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:31.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:30.823+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:31.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:30.823+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:31.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:30.823+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:31.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:30.823+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:31.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:31.023+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:31.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:31.023+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:31.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:31.023+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:31.177 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:31.024+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:32.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:31.852+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:32.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:31.852+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:32.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:31.852+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:32.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:31.853+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:32.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:31.816+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:32.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:31.816+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:32.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:31.816+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:32.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:31.817+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:32.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:32.026+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:32.177 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:32.026+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:32.177 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:32.026+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:32.177 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:32.027+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:33.047 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:32.806+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:33.047 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:32.806+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:33.047 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:32.806+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:33.047 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:32.807+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:33.048 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:32.794+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:33.048 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:32.794+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:33.048 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:32.794+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:33.048 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:32.795+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:33.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:33.044+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:33.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:33.044+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:33.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:33.044+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:33.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:33.045+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:33.830+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:33.830+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:33.830+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:33.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:33.831+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:33.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:33.834+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:33.927 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:33.834+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:33.927 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:33.834+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:33.927 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:33.835+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:34.359 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:41:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:41:34.010+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22432 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:41:34.359 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:34.094+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:34.360 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:34.094+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:34.360 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:34.094+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:34.360 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:34.095+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:35.124 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:34.801+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:35.124 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:34.801+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:35.124 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:34.801+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:35.124 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:34.802+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:35.124 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:34.791+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:35.124 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:34.791+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:35.125 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:34.791+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:35.125 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:34.791+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:35.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:35.121+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:35.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:35.121+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:35.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:35.121+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:35.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:35.122+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:36.113 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:35.838+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:36.113 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:35.838+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:36.114 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:35.838+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:36.114 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:35.839+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:36.114 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:35.789+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:36.114 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:35.789+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:36.114 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:35.789+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:36.114 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:35.790+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:36.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:36.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:36.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:36.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:36.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:36.111+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:36.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:36.112+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:36.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:36.811+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:36.811+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:36.811+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:36.812+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:36.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:36.830+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:36.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:36.830+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:36.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:36.830+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:36.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:36.831+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:37.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:37.073+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:37.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:37.073+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:37.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:37.073+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:37.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:37.073+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:38.077 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:37.858+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:38.077 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:37.858+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:38.077 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:37.858+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:38.078 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:37.859+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:38.078 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:37.849+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:38.078 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:37.849+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:38.078 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:37.849+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:38.078 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:37.849+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:38.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:38.074+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:38.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:38.074+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:38.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:38.074+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:38.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:38.075+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:39.098 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:41:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:41:39.013+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22485 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:41:39.098 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:38.875+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:39.098 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:38.875+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:39.098 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:38.875+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:39.099 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:38.876+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:39.099 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:38.836+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:39.099 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:38.836+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:39.099 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:38.836+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:39.099 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:38.837+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:39.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:39.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:39.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:39.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:39.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:39.095+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:39.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:39.096+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:40.140 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:39.813+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:40.140 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:39.813+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:40.141 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:39.813+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:40.141 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:39.815+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:40.141 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:39.899+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:40.141 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:39.899+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:40.141 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:39.899+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:40.142 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:39.900+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:40.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:40.138+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:40.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:40.138+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:40.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:40.138+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:40.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:40.139+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:41.131 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:40.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:41.131 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:40.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:41.131 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:40.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:41.131 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:40.769+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:41.132 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:40.894+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:41.132 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:40.894+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:41.132 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:40.894+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:41.132 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:40.894+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:41.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:41.128+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:41.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:41.128+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:41.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:41.128+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:41.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:41.129+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:41.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:41.765+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:41.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:41.766+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:41.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:41.766+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:41.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:41.767+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:42.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:42.084+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:42.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:42.084+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:42.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:42.084+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:42.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:42.085+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:42.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:41.939+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:42.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:41.939+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:42.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:41.939+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:42.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:41.940+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:43.132 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:42.794+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:43.132 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:42.794+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:43.132 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:42.794+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:43.133 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:42.795+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:43.133 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:42.970+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:43.133 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:42.970+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:43.133 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:42.970+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:43.133 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:42.971+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:43.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:43.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:43.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:43.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:43.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:43.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:43.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:43.131+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:43.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:43.825+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:43.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:43.825+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:43.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:43.825+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:43.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:43.826+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:44.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:41:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:41:44.015+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22537 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:41:44.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:43.928+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:44.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:43.928+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:44.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:43.928+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:44.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:43.929+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:44.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:44.091+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:44.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:44.091+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:44.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:44.091+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:44.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:44.092+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:45.128 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:44.939+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:45.129 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:44.939+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:45.129 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:44.939+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:45.129 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:44.940+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:45.129 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:44.802+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:45.129 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:44.802+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:45.130 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:44.802+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:45.130 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:44.802+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:45.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:45.126+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:45.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:45.126+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:45.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:45.126+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:45.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:45.127+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:46.086 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:45.786+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:46.087 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:45.786+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:46.087 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:45.786+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:46.087 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:45.787+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:46.087 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:45.975+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:46.087 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:45.975+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:46.087 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:45.975+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:46.088 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:45.976+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:46.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:46.084+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:46.084+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:46.084+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:46.085+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:46.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:46.782+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:46.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:46.782+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:46.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:46.782+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:46.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:46.783+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:47.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:47.006+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:47.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:47.006+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:47.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:47.006+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:47.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:47.007+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:47.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:47.090+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:47.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:47.090+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:47.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:47.090+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:47.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:47.091+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:47.999 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:47.735+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:47.999 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:47.735+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:47.999 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:47.735+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:47.999 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:47.736+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:48.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:47.996+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:48.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:47.996+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:48.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:47.996+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:48.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:47.997+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:48.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:48.104+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:48.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:48.105+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:48.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:48.105+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:48.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:48.105+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:49.047 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:41:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:41:49.017+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22589 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:41:49.047 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:48.772+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:49.047 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:48.772+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:49.048 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:48.772+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:49.048 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:48.773+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:49.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:49.097+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:49.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:49.097+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:49.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:49.097+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:49.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:49.097+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:49.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:49.044+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:49.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:49.044+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:49.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:49.044+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:49.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:49.045+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:50.044 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:49.731+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:50.045 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:49.731+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:50.045 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:49.731+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:50.045 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:49.732+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:50.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:50.100+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:50.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:50.100+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:50.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:50.100+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:50.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:50.101+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:50.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:50.042+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:50.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:50.042+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:50.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:50.042+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:50.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:50.043+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:51.050 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:50.721+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:51.050 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:50.721+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:51.050 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:50.721+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:51.050 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:50.722+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:51.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:51.142+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:51.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:51.142+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:51.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:51.142+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:51.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:51.143+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:51.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:51.047+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:51.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:51.047+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:51.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:51.047+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:51.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:51.048+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:51.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:51.725+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:51.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:51.725+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:51.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:51.725+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:51.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:51.725+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:52.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:52.183+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:52.183+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:52.183+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:52.184+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:52.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:52.024+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:52.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:52.024+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:52.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:52.024+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:52.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:52.025+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:53.061 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:52.735+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:53.062 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:52.735+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:53.062 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:52.735+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:53.062 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:52.736+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:53.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:53.154+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:53.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:53.154+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:53.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:53.154+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:53.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:53.155+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:53.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:53.059+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:53.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:53.059+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:53.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:53.059+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:53.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:53.060+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:53.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:53.770+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:53.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:53.770+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:53.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:53.770+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:53.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:53.770+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:54.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:41:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:41:54.019+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22641 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:41:54.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:54.197+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:54.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:54.197+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:54.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:54.197+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:54.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:54.198+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:54.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:54.056+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:54.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:54.056+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:54.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:54.056+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:54.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:54.057+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:55.056 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:54.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:55.056 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:54.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:55.057 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:54.769+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:55.057 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:54.770+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:55.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:55.187+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:55.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:55.187+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:55.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:55.187+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:55.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:55.188+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:55.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:55.054+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:55.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:55.054+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:55.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:55.054+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:55.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:55.055+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:56.068 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:55.775+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:56.068 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:55.775+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:56.068 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:55.775+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:56.068 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:55.776+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:56.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:56.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:56.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:56.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:56.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:56.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:56.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:56.215+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:56.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:56.065+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:56.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:56.065+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:56.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:56.065+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:56.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:56.066+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:57.043 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:56.775+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:57.044 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:56.775+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:57.044 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:56.775+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:57.044 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:56.776+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:57.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:57.041+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:57.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:57.041+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:57.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:57.041+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:57.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:57.042+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:57.261+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:57.261+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:57.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:57.261+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:57.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:57.262+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:58.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:57.806+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:58.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:57.806+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:58.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:57.806+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:58.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:57.807+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:58.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:58.018+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:58.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:58.018+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:58.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:58.018+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:58.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:58.019+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:41:58.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:58.223+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:58.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:58.223+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:58.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:58.223+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:58.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:58.224+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:59.071 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:41:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:41:59.022+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22694 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:41:59.071 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:58.797+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:59.071 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:58.797+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:59.071 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:58.797+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:41:59.072 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:58.798+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:59.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:59.239+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:59.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:59.239+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:59.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:59.239+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:41:59.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:41:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:41:59.240+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:41:59.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:59.068+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:59.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:59.068+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:59.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:59.068+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:41:59.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:41:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:41:59.070+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:00.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:59.833+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:00.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:59.833+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:00.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:59.833+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:00.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:41:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:41:59.834+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:00.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:00.060+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:00.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:00.060+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:00.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:00.060+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:00.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:00.061+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:00.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:00.210+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:00.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:00.210+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:00.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:00.210+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:00.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:00.212+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:01.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:00.869+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:01.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:00.869+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:01.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:00.869+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:01.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:00.870+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:01.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:01.029+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:01.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:01.029+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:01.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:01.029+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:01.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:01.030+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:01.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:01.219+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:01.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:01.219+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:01.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:01.219+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:01.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:01.220+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:02.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:01.887+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:02.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:01.887+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:02.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:01.887+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:02.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:01.888+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:02.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:02.035+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:02.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:02.035+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:02.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:02.035+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:02.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:02.036+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:02.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:02.260+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:02.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:02.260+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:02.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:02.260+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:02.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:02.260+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:03.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:02.919+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:03.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:02.919+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:03.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:02.919+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:03.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:02.919+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:03.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:03.075+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:03.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:03.075+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:03.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:03.075+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:03.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:03.076+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:03.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:03.262+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:03.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:03.263+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:03.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:03.263+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:03.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:03.264+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:04.216 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:42:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:42:04.024+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22746 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:42:04.217 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:03.923+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:04.217 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:03.923+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:04.217 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:03.923+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:04.217 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:03.924+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:04.218 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:04.117+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:04.218 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:04.117+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:04.218 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:04.117+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:04.218 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:04.118+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:04.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:04.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:04.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:04.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:04.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:04.214+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:04.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:04.215+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:05.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:05.101+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:05.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:05.101+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:05.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:05.101+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:05.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:05.102+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:05.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:05.182+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:05.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:05.182+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:05.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:05.182+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:05.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:05.183+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:05.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:04.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:05.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:04.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:05.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:04.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:05.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:04.962+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:06.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:06.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:06.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:06.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:06.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:06.164+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:06.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:06.165+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:06.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:06.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:06.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:06.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:06.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:06.000+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:06.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:06.001+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:06.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:06.052+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:06.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:06.052+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:06.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:06.052+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:06.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:06.053+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:07.087 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:06.960+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:07.087 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:06.960+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:07.088 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:06.960+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:07.088 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:06.961+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:07.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:07.215+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:07.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:07.215+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:07.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:07.215+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:07.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:07.215+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:07.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:07.085+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:07.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:07.085+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:07.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:07.085+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:07.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:07.086+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:08.259 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:08.080+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:08.259 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:08.080+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:08.259 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:08.080+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:08.260 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:08.080+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:08.260 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:07.925+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:08.260 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:07.925+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:08.260 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:07.925+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:08.260 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:07.926+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:08.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:08.256+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:08.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:08.256+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:08.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:08.256+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:08.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:08.256+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:09.299 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:42:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:42:09.025+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22799 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:42:09.300 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:08.952+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:09.300 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:08.952+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:09.300 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:08.952+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:09.300 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:08.953+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:09.301 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:09.050+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:09.301 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:09.050+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:09.301 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:09.050+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:09.301 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:09.051+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:09.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:09.296+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:09.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:09.296+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:09.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:09.296+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:09.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:09.297+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:10.284 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:09.979+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:10.285 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:09.979+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:10.285 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:09.979+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:10.285 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:09.979+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:10.285 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:10.045+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:10.285 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:10.045+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:10.286 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:10.045+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:10.286 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:10.046+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:10.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:10.281+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:10.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:10.281+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:10.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:10.281+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:10.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:10.282+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:11.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:11.249+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:11.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:11.249+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:11.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:11.249+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:11.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:11.249+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:11.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:11.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:11.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:11.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:11.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:11.016+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:11.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:11.016+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:11.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:11.025+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:11.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:11.025+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:11.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:11.025+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:11.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:11.026+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:12.174 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:12.002+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:12.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:12.002+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:12.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:12.002+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:12.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:12.003+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:12.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:12.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:12.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:12.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:12.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:12.050+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:12.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:12.051+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:12.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:12.247+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:12.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:12.247+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:12.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:12.247+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:12.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:12.247+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:13.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:13.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:13.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:13.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:13.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:13.033+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:13.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:13.033+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:13.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:13.207+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:13.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:13.207+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:13.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:13.207+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:13.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:13.208+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:13.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:12.996+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:13.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:12.996+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:13.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:12.996+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:13.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:12.997+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:14.257 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:42:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:42:14.027+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22852 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:42:14.258 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:14.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:14.258 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:14.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:14.258 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:14.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:14.258 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:14.053+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:14.259 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:13.965+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:14.259 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:13.965+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:14.259 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:13.965+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:14.259 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:13.965+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:14.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:14.254+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:14.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:14.254+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:14.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:14.254+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:14.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:14.255+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:15.256 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:15.002+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:15.256 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:15.002+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:15.256 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:15.002+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:15.256 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:15.003+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:15.256 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:15.008+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:15.257 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:15.008+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:15.257 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:15.008+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:15.257 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:15.009+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:15.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:15.252+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:15.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:15.252+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:15.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:15.252+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:15.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:15.253+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:16.269 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:16.022+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:16.270 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:16.022+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:16.270 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:16.022+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:16.270 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:16.023+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:16.270 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:15.965+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:16.270 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:15.965+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:16.270 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:15.965+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:16.271 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:15.966+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:16.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:16.266+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:16.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:16.266+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:16.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:16.266+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:16.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:16.267+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:17.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:16.993+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:17.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:16.993+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:17.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:16.993+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:17.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:16.994+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:17.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:17.071+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:17.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:17.071+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:17.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:17.071+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:17.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:17.072+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:17.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:17.299+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:17.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:17.299+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:17.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:17.299+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:17.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:17.299+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:18.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:18.018+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:18.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:18.018+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:18.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:18.018+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:18.352 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:18.018+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:18.353 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:18.077+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:18.353 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:18.077+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:18.353 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:18.077+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:18.353 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:18.078+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:18.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:18.349+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:18.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:18.349+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:18.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:18.349+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:18.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:18.350+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:19.384 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:42:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:42:19.030+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22905 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:42:19.385 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:19.009+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:19.385 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:19.009+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:19.385 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:19.009+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:19.385 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:19.009+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:19.386 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:19.070+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:19.386 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:19.070+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:19.386 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:19.070+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:19.386 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:19.071+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:19.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:19.382+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:19.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:19.382+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:19.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:19.382+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:19.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:19.382+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:20.407 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:19.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:20.408 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:19.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:20.408 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:19.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:20.408 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:19.998+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:20.408 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:20.096+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:20.408 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:20.096+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:20.408 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:20.096+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:20.409 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:20.096+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:20.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:20.405+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:20.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:20.405+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:20.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:20.405+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:20.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:20.405+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:21.390 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:21.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:21.390 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:21.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:21.390 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:21.002+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:21.390 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:21.002+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:21.391 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:21.061+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:21.391 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:21.061+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:21.391 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:21.061+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:21.391 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:21.063+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:21.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:21.386+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:21.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:21.386+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:21.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:21.386+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:21.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:21.388+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:22.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:22.069+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:22.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:22.069+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:22.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:22.069+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:22.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:22.070+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:22.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:21.999+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:22.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:21.999+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:22.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:21.999+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:22.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:22.000+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:22.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:22.438+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:22.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:22.438+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:22.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:22.438+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:22.925 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:22.439+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:23.409 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:23.085+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:23.409 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:23.085+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:23.409 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:23.085+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:23.409 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:23.086+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:23.410 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:23.019+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:23.410 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:23.019+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:23.410 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:23.019+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:23.410 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:23.020+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:23.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:23.406+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:23.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:23.406+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:23.673 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:23.406+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:23.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:23.407+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:24.418 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:42:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:42:24.033+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 22957 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:42:24.418 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:23.977+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:24.418 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:23.978+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:24.418 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:23.978+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:24.419 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:23.979+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:24.419 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:24.125+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:24.419 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:24.125+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:24.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:24.125+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:24.420 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:24.126+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:24.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:24.415+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:24.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:24.415+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:24.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:24.415+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:24.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:24.416+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:25.376 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:24.938+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:25.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:24.938+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:25.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:24.938+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:25.377 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:24.939+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:25.377 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:25.164+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:25.377 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:25.164+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:25.377 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:25.164+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:25.378 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:25.165+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:25.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:25.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:25.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:25.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:25.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:25.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:25.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:25.375+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:26.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:25.910+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:26.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:25.910+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:26.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:25.910+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:26.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:25.911+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:26.583 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:26.421+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:26.584 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:26.421+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:26.584 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:26.421+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:26.584 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:26.422+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:26.584 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:26.187+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:26.584 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:26.187+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:26.585 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:26.187+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:26.585 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:26.188+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:27.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:26.906+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:27.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:26.906+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:27.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:26.906+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:27.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:26.907+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:27.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:27.426+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:27.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:27.426+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:27.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:27.426+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:27.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:27.427+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:27.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:27.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:27.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:27.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:27.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:27.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:27.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:27.237+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:28.235 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:27.944+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:28.235 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:27.944+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:28.235 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:27.944+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:28.235 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:27.945+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:28.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:28.438+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:28.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:28.438+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:28.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:28.438+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:28.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:28.439+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:28.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:28.232+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:28.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:28.232+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:28.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:28.232+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:28.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:28.234+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:29.174 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:42:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:42:29.035+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23010 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:42:29.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:28.910+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:29.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:28.910+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:29.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:28.910+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:29.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:28.911+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:29.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:29.260+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:29.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:29.260+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:29.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:29.260+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:29.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:29.261+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:29.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:29.487+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:29.677 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:29.487+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:29.677 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:29.487+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:29.677 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:29.488+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:30.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:29.906+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:30.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:29.906+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:30.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:29.906+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:30.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:29.907+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:30.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:30.463+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:30.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:30.463+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:30.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:30.464+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:30.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:30.464+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:30.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:30.291+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:30.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:30.291+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:30.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:30.291+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:30.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:30.292+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:31.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:30.910+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:31.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:30.910+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:31.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:30.910+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:31.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:30.911+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:31.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:31.417+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:31.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:31.417+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:31.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:31.417+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:31.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:31.418+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:31.572 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:31.268+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:31.572 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:31.268+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:31.572 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:31.268+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:31.572 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:31.269+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:32.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:31.906+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:32.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:31.906+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:32.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:31.906+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:32.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:31.907+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:32.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:32.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:32.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:32.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:32.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:32.465+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:32.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:32.465+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:32.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:32.307+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:32.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:32.307+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:32.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:32.307+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:32.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:32.308+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:33.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:32.866+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:33.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:32.866+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:33.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:32.866+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:33.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:32.867+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:33.673 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:33.316+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:33.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:33.316+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:33.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:33.316+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:33.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:33.317+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:33.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:33.470+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:33.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:33.470+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:33.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:33.470+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:33.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:33.471+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:33.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:33.831+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:33.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:33.831+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:33.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:33.831+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:33.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:33.831+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:34.338 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:42:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:42:34.037+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23062 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:42:34.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:34.475+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:34.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:34.475+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:34.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:34.475+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:34.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:34.476+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:34.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:34.335+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:34.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:34.335+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:34.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:34.335+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:34.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:34.336+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:35.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:34.840+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:35.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:34.840+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:35.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:34.840+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:35.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:34.841+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:35.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:35.378+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:35.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:35.378+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:35.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:35.378+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:35.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:35.379+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:35.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:35.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:35.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:35.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:35.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:35.460+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:35.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:35.461+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:36.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:35.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:36.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:35.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:36.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:35.872+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:36.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:35.873+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:36.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:36.427+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:36.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:36.427+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:36.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:36.427+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:36.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:36.428+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:36.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:36.416+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:36.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:36.416+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:36.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:36.416+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:36.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:36.417+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:37.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:36.917+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:37.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:36.917+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:37.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:36.917+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:37.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:36.917+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:37.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:37.382+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:37.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:37.382+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:37.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:37.382+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:37.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:37.383+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:37.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:37.461+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:37.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:37.461+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:37.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:37.461+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:37.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:37.462+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:38.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:37.930+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:38.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:37.930+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:38.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:37.930+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:38.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:37.931+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:38.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:38.381+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:38.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:38.381+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:38.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:38.381+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:38.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:38.383+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:38.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:38.446+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:38.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:38.446+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:38.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:38.446+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:38.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:38.446+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:39.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:42:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:42:39.040+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23115 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:42:39.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:38.896+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:39.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:38.896+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:39.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:38.896+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:39.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:38.897+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:39.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:39.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:39.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:39.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:39.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:39.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:39.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:39.378+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:39.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:39.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:39.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:39.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:39.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:39.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:39.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:39.449+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:40.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:39.890+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:40.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:39.890+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:40.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:39.890+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:40.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:39.890+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:40.378 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:40.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:40.378 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:40.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:40.378 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:40.374+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:40.378 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:40.375+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:40.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:40.402+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:40.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:40.402+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:40.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:40.402+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:40.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:40.403+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:41.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:40.881+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:41.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:40.881+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:41.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:40.881+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:41.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:40.882+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:41.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:41.402+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:41.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:41.402+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:41.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:41.402+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:41.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:41.403+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:41.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:41.443+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:41.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:41.443+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:41.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:41.443+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:41.677 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:41.444+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:42.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:41.905+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:42.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:41.905+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:42.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:41.905+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:42.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:41.905+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:42.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:42.425+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:42.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:42.425+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:42.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:42.425+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:42.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:42.426+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:42.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:42.402+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:42.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:42.402+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:42.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:42.402+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:42.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:42.403+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:43.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:42.867+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:43.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:42.867+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:43.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:42.867+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:43.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:42.868+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:43.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:43.468+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:43.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:43.468+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:43.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:43.468+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:43.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:43.469+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:43.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:43.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:43.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:43.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:43.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:43.377+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:43.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:43.378+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:43.924 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:43.859+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:43.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:43.859+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:43.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:43.859+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:43.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:43.860+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:44.333 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:42:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:42:44.042+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23167 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:42:44.674 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:44.330+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:44.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:44.330+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:44.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:44.330+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:44.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:44.332+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:44.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:44.432+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:44.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:44.432+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:44.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:44.432+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:44.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:44.433+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:45.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:44.871+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:45.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:44.871+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:45.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:44.871+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:45.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:44.872+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:45.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:45.287+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:45.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:45.287+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:45.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:45.287+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:45.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:45.288+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:45.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:45.432+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:45.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:45.432+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:45.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:45.432+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:45.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:45.433+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:46.174 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:45.921+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:46.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:45.921+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:46.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:45.921+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:46.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:45.922+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:46.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:46.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:46.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:46.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:46.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:46.448+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:46.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:46.449+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:46.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:46.249+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:46.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:46.249+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:46.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:46.249+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:46.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:46.250+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:47.215 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:46.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:47.215 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:46.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:47.215 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:46.961+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:47.215 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:46.961+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:47.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:47.446+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:47.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:47.446+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:47.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:47.446+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:47.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:47.447+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:47.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:47.212+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:47.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:47.212+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:47.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:47.212+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:47.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:47.213+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:48.414 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:48.177+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:48.414 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:48.177+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:48.414 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:48.177+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:48.414 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:48.177+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:48.415 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:48.006+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:48.415 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:48.006+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:48.415 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:48.006+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:48.415 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:48.007+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:48.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:48.411+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:48.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:48.411+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:48.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:48.411+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:48.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:48.412+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:49.379 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:42:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:42:49.044+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23220 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:42:49.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:49.142+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:49.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:49.142+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:49.380 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:49.142+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:49.380 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:49.143+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:49.380 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:49.376+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:49.380 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:49.376+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:49.380 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:49.376+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:49.381 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:48.994+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:49.381 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:48.994+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:49.381 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:48.994+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:49.381 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:48.995+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:49.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:49.377+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:50.354 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:50.165+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:50.354 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:50.165+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:50.354 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:50.165+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:50.354 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:50.166+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:50.355 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:50.012+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:50.355 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:50.012+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:50.355 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:50.012+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:50.355 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:50.013+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:50.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:50.351+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:50.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:50.351+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:50.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:50.351+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:50.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:50.352+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:51.378 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:51.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:51.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:51.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:51.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:51.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:51.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:51.142+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:51.379 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:51.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:51.379 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:51.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:51.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:51.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:51.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:51.037+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:51.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:51.375+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:51.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:51.375+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:51.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:51.375+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:51.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:51.376+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:52.340 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:52.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:52.340 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:52.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:52.340 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:52.077+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:52.341 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:52.078+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:52.341 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:52.162+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:52.341 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:52.162+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:52.342 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:52.162+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:52.342 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:52.162+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:52.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:52.337+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:52.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:52.337+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:52.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:52.337+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:52.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:52.338+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:53.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:53.158+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:53.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:53.158+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:53.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:53.158+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:53.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:53.158+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:53.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:53.296+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:53.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:53.296+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:53.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:53.296+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:53.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:53.297+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:53.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:53.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:53.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:53.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:53.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:53.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:53.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:53.087+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:54.310 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:42:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:42:54.047+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23272 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:42:54.311 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:54.106+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:54.311 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:54.106+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:54.311 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:54.106+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:54.311 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:54.107+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:54.311 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:54.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:54.311 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:54.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:54.312 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:54.141+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:54.312 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:54.142+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:54.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:54.308+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:54.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:54.308+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:54.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:54.308+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:54.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:54.309+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:55.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:55.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:55.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:55.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:55.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:55.130+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:55.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:55.132+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:55.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:55.289+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:55.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:55.289+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:55.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:55.289+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:55.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:55.290+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:55.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:55.138+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:55.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:55.138+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:55.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:55.138+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:55.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:55.138+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:56.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:56.088+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:56.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:56.088+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:56.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:56.088+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:56.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:56.089+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:56.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:56.106+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:56.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:56.106+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:56.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:56.106+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:56.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:56.107+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:56.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:56.244+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:56.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:56.244+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:56.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:56.244+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:56.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:56.246+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:57.137+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:57.137+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:57.137+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:57.137+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:57.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:57.247+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:57.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:57.247+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:57.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:57.247+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:57.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:57.248+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:57.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:57.080+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:57.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:57.080+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:57.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:57.080+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:57.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:57.081+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:58.424 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:58.177+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:58.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:58.177+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:58.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:58.177+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:58.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:58.177+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:58.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:58.273+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:58.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:58.273+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:58.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:58.273+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:58.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:58.274+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:58.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:58.127+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:58.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:58.127+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:58.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:58.127+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:58.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:58.127+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:59.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:42:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:42:59.049+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23325 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:42:59.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:59.128+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:59.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:59.128+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:59.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:59.128+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:42:59.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:42:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:42:59.129+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:42:59.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:59.259+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:59.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:59.259+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:59.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:59.259+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:42:59.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:42:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:42:59.260+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:42:59.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:59.195+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:59.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:59.195+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:59.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:59.195+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:42:59.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:42:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:42:59.196+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:00.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:00.153+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:00.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:00.153+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:00.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:00.153+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:00.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:00.154+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:00.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:00.116+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:00.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:00.116+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:00.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:00.116+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:00.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:00.117+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:00.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:00.221+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:00.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:00.221+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:00.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:00.221+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:00.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:00.222+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:01.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:01.153+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:01.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:01.153+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:01.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:01.153+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:01.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:01.154+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:01.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:01.131+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:01.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:01.131+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:01.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:01.131+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:01.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:01.131+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:01.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:01.235+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:01.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:01.235+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:01.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:01.235+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:01.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:01.236+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:02.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:02.191+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:02.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:02.191+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:02.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:02.191+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:02.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:02.192+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:02.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:02.175+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:02.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:02.175+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:02.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:02.175+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:02.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:02.176+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:02.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:02.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:02.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:02.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:02.677 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:02.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:02.677 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:02.237+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:03.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:03.193+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:03.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:03.193+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:03.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:03.193+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:03.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:03.193+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:03.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:03.234+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:03.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:03.234+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:03.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:03.234+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:03.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:03.235+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:03.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:03.146+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:03.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:03.146+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:03.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:03.146+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:03.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:03.147+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:04.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:43:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:43:04.051+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23377 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:43:04.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:04.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:04.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:04.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:04.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:04.236+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:04.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:04.236+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:04.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:04.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:04.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:04.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:04.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:04.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:04.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:04.147+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:04.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:04.167+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:04.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:04.167+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:04.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:04.167+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:04.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:04.167+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:05.424 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:05.240+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:05.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:05.240+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:05.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:05.240+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:05.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:05.241+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:05.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:05.157+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:05.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:05.157+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:05.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:05.157+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:05.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:05.158+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:05.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:05.123+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:05.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:05.123+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:05.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:05.123+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:05.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:05.124+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:06.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:06.160+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:06.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:06.160+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:06.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:06.160+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:06.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:06.160+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:06.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:06.247+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:06.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:06.247+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:06.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:06.247+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:06.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:06.248+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:06.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:06.077+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:06.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:06.077+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:06.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:06.077+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:06.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:06.078+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:07.303 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:07.252+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:07.303 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:07.252+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:07.303 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:07.252+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:07.304 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:07.253+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:07.304 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:07.036+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:07.304 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:07.036+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:07.304 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:07.036+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:07.304 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:07.037+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:07.305 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:07.196+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:07.305 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:07.196+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:07.305 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:07.196+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:07.305 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:07.197+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:08.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:08.229+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:08.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:08.229+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:08.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:08.229+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:08.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:08.231+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:08.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:08.036+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:08.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:08.036+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:08.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:08.036+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:08.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:08.037+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:08.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:08.177+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:08.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:08.177+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:08.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:08.177+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:08.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:08.178+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:09.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:43:09.054+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23430 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:43:09.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:09.244+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:09.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:09.244+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:09.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:09.244+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:09.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:09.245+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:09.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:09.208+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:09.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:09.208+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:09.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:09.208+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:09.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:09.209+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:09.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:09.005+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:09.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:09.005+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:09.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:09.005+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:09.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:09.006+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:10.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:09.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:10.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:09.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:10.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:09.979+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:10.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:09.980+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:10.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:10.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:10.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:10.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:10.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:10.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:10.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:10.218+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:10.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:10.198+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:10.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:10.198+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:10.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:10.198+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:10.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:10.199+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:11.231 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:10.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:11.232 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:10.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:11.232 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:10.955+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:11.232 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:10.956+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:11.573 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:11.235+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:11.573 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:11.235+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:11.573 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:11.235+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:11.573 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:11.236+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:11.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:11.229+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:11.574 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:11.229+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:11.574 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:11.229+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:11.574 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:11.230+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:12.346 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:12.250+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:12.347 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:12.250+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:12.347 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:12.250+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:12.347 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:12.251+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:12.347 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:12.223+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:12.348 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:12.223+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:12.348 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:12.223+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:12.348 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:12.224+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:12.348 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:12.006+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:12.348 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:12.006+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:12.348 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:12.006+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:12.349 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:12.007+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:13.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:13.221+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:13.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:13.221+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:13.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:13.221+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:13.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:13.222+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:13.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:13.187+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:13.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:13.187+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:13.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:13.187+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:13.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:13.188+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:13.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:13.005+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:13.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:13.005+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:13.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:13.005+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:13.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:13.006+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:14.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:43:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:43:14.056+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23482 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:43:14.240 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:14.222+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:14.241 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:14.222+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:14.241 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:14.222+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:14.241 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:14.222+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:14.241 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:13.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:14.241 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:13.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:14.241 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:13.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:14.242 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:13.979+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:14.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:14.238+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:14.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:14.238+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:14.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:14.238+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:14.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:14.239+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:15.287 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:14.946+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:15.288 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:14.946+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:15.288 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:14.946+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:15.288 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:14.946+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:15.288 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:15.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:15.288 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:15.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:15.288 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:15.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:15.289 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:15.177+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:15.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:15.285+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:15.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:15.285+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:15.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:15.285+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:15.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:15.286+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:16.312 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:15.996+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:16.313 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:15.996+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:16.313 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:15.996+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:16.313 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:15.997+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:16.313 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:16.213+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:16.313 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:16.213+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:16.313 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:16.213+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:16.314 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:16.213+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:16.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:16.310+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:16.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:16.310+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:16.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:16.310+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:16.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:16.311+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:17.333 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:17.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:17.334 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:17.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:17.334 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:17.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:17.334 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:17.040+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:17.334 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:17.204+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:17.334 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:17.204+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:17.334 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:17.204+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:17.335 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:17.205+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:17.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:17.331+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:17.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:17.331+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:17.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:17.331+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:17.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:17.332+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:18.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:18.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:18.340 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:18.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:18.341 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:18.083+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:18.341 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:18.083+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:18.341 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:18.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:18.341 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:18.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:18.341 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:18.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:18.341 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:18.222+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:18.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:18.338+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:18.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:18.338+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:18.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:18.338+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:18.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:18.339+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:19.360 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:43:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:43:19.059+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23535 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:43:19.361 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:19.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:19.361 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:19.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:19.361 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:19.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:19.361 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:19.048+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:19.361 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:19.255+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:19.362 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:19.255+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:19.362 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:19.255+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:19.362 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:19.256+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:19.362 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:19.357+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:19.362 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:19.358+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:19.363 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:19.358+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:19.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:19.358+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:20.363 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:20.069+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:20.363 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:20.069+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:20.363 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:20.069+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:20.363 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:20.070+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:20.364 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:20.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:20.364 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:20.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:20.364 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:20.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:20.364 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:20.294+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:20.364 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:20.360+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:20.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:20.360+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:20.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:20.360+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:20.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:20.361+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:21.356 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:21.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:21.356 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:21.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:21.356 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:21.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:21.357 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:21.273+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:21.357 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:21.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:21.357 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:21.047+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:21.357 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:21.048+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:21.357 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:21.048+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:21.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:21.353+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:21.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:21.353+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:21.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:21.353+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:21.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:21.355+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:22.347 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:22.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:22.347 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:22.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:22.347 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:22.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:22.348 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:22.040+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:22.348 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:22.262+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:22.348 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:22.262+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:22.348 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:22.263+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:22.348 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:22.263+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:22.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:22.344+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:22.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:22.344+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:22.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:22.344+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:22.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:22.346+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:23.377 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:23.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:23.377 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:23.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:23.378 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:23.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:23.378 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:23.040+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:23.378 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:23.224+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:23.378 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:23.224+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:23.379 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:23.224+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:23.379 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:23.225+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:23.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:23.375+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:23.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:23.375+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:23.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:23.375+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:23.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:23.376+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:24.417 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:43:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:43:24.061+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23587 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:43:24.418 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:24.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:24.418 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:24.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:24.418 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:24.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:24.419 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:24.047+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:24.419 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:24.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:24.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:24.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:24.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:24.221+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:24.420 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:24.222+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:24.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:24.415+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:24.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:24.415+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:24.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:24.415+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:24.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:24.416+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:25.211 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:25.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:25.211 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:25.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:25.211 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:25.022+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:25.211 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:25.023+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:25.674 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:25.208+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:25.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:25.208+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:25.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:25.208+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:25.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:25.209+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:25.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:25.380+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:25.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:25.380+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:25.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:25.380+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:25.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:25.381+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:26.390 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:26.028+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:26.390 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:26.028+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:26.390 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:26.028+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:26.391 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:26.029+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:26.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:26.161+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:26.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:26.161+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:26.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:26.161+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:26.391 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:26.162+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:26.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:26.387+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:26.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:26.387+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:26.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:26.387+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:26.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:26.389+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:27.402 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:27.008+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:27.402 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:27.008+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:27.402 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:27.008+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:27.402 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:27.008+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:27.403 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:27.129+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:27.403 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:27.129+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:27.403 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:27.129+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:27.403 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:27.129+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:27.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:27.399+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:27.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:27.399+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:27.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:27.399+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:27.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:27.400+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:28.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:27.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:28.379 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:27.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:28.380 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:27.975+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:28.380 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:27.977+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:28.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:28.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:28.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:28.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:28.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:28.175+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:28.380 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:28.176+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:28.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:28.377+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:28.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:28.377+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:28.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:28.377+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:28.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:28.377+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:29.090 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-04-27T04:43:29.095 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T04:43:29.095 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-04-27T04:43:29.096 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-04-27T04:43:29.108 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-27T04:43:29.119 INFO:tasks.cephadm:Teardown begin 2024-04-27T04:43:29.119 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephadm.py", line 2246, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-04-27T04:43:29.124 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T04:43:29.155 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:43:29.156 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:43:29.403 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:43:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:43:29.063+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23640 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:43:29.403 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:28.982+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:29.403 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:28.982+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:29.403 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:28.982+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:29.404 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:28.983+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:29.404 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:29.196+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:29.404 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:29.196+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:29.404 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:29.196+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:29.404 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:29.198+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:29.674 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:29.400+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:29.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:29.400+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:29.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:29.400+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:29.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:29.401+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:30.248 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:29.983+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:30.249 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:29.983+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:30.249 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:29.983+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:30.249 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:29.983+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:30.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:30.404+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:30.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:30.404+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:30.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:30.404+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:30.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:30.405+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:30.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:30.246+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:30.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:30.246+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:30.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:30.246+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:30.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:30.246+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:31.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:31.200+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:31.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:31.200+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:31.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:31.200+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:31.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:31.201+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:31.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:31.021+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:31.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:31.021+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:31.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:31.021+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:31.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:31.022+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:31.695 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:31.444+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:31.696 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:31.444+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:31.696 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:31.444+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:31.696 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:31.445+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:32.226 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:43:32.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:32.188+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:32.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:32.188+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:32.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:32.188+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:32.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:32.188+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:32.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:32.025+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:32.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:32.025+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:32.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:32.025+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:32.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:32.026+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:32.447+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:32.447+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:32.447+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:32.448+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:33.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:33.233+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:33.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:33.233+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:33.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:33.233+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:33.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:33.234+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:33.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:33.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:33.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:33.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:33.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:33.062+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:33.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:33.063+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:33.484+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:33.484+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:33.484+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:33.485+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:34.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:43:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:43:34.066+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23692 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:43:34.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:34.227+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:34.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:34.227+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:34.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:34.227+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:34.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:34.228+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:34.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:34.050+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:34.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:34.050+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:34.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:34.050+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:34.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:34.051+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:34.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:34.496+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:34.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:34.496+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:34.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:34.496+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:34.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:34.498+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:35.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:35.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:35.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:35.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:35.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:35.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:35.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:35.047+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:35.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:35.235+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:35.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:35.235+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:35.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:35.235+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:35.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:35.236+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:35.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:35.529+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:35.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:35.529+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:35.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:35.529+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:35.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:35.531+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:36.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:36.253+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:36.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:36.253+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:36.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:36.253+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:36.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:36.254+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:36.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:36.072+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:36.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:36.072+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:36.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:36.072+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:36.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:36.072+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:36.493+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:36.493+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:36.493+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:36.494+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:37.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:37.301+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:37.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:37.301+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:37.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:37.301+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:37.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:37.301+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:37.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:37.061+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:37.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:37.061+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:37.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:37.061+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:37.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:37.062+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:37.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:37.533+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:37.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:37.533+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:37.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:37.533+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:37.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:37.533+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:38.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:38.256+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:38.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:38.256+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:38.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:38.256+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:38.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:38.257+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:38.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:38.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:38.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:38.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:38.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:38.039+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:38.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:38.039+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:38.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:38.575+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:38.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:38.575+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:38.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:38.575+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:38.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:38.576+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:39.292 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:43:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:43:39.068+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23745 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:43:39.293 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:39.020+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:39.293 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:39.020+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:39.293 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:39.020+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:39.293 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:39.021+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:39.581 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:39.290+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:39.581 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:39.290+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:39.581 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:39.290+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:39.582 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:39.290+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:39.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:39.578+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:39.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:39.578+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:39.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:39.578+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:39.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:39.579+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:40.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:40.269+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:40.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:40.269+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:40.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:40.269+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:40.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:40.269+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:40.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:40.048+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:40.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:40.048+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:40.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:40.048+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:40.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:40.049+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:40.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:40.604+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:40.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:40.605+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:40.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:40.605+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:40.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:40.605+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:41.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:43:41.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:43:41.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:41.049+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:41.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:41.049+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:41.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:41.049+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:41.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:41.049+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:41.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:41.292+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:41.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:41.292+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:41.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:41.292+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:41.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:41.293+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:41.615+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:41.615+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:41.615+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:41.616+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:42.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:42.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:42.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:42.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:42.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:42.308+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:42.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:42.309+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:42.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:42.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:42.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:42.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:42.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:42.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:42.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:42.083+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:42.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:42.577+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:42.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:42.577+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:42.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:42.577+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:42.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:42.578+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:43.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:43.132+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:43.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:43.132+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:43.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:43.132+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:43.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:43.133+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:43.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:43.335+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:43.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:43.335+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:43.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:43.335+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:43.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:43.336+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:43.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:43.611+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:43.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:43.611+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:43.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:43.611+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:43.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:43.612+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:44.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:43:44.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:43:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:43:44.069+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23797 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:43:44.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:44.144+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:44.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:44.144+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:44.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:44.144+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:44.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:44.145+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:44.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:44.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:44.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:44.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:44.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:44.310+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:44.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:44.311+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:44.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:44.601+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:44.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:44.601+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:44.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:44.601+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:44.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:44.602+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:45.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:45.142+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:45.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:45.142+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:45.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:45.142+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:45.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:45.143+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:45.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:45.350+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:45.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:45.350+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:45.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:45.350+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:45.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:45.351+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:45.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:45.622+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:45.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:45.622+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:45.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:45.622+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:45.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:45.623+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:46.124+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:46.124+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:46.124+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:46.124+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:46.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:46.353+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:46.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:46.353+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:46.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:46.353+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:46.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:46.354+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:46.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:46.621+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:46.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:46.621+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:46.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:46.621+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:46.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:46.622+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:47.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:47.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:47.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:47.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:47.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:47.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:47.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:47.148+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:47.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:47.328+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:47.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:47.328+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:47.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:47.328+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:47.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:47.329+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:47.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:47.593+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:47.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:47.593+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:47.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:47.593+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:47.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:47.594+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:48.607 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:48.192+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:48.607 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:48.192+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:48.607 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:48.192+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:48.607 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:48.193+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:48.608 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:48.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:48.608 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:48.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:48.608 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:48.344+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:48.608 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:48.345+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:48.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:48.603+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:48.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:48.603+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:48.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:48.603+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:48.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:48.605+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:49.333 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:43:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:43:49.072+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23850 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:43:49.333 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:49.194+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:49.333 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:49.194+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:49.333 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:49.194+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:49.334 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:49.194+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:49.631 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:49.329+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:49.632 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:49.329+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:49.632 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:49.329+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:49.632 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:49.330+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:49.628+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:49.628+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:49.628+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:49.629+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:50.669 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:50.231+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:50.669 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:50.231+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:50.669 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:50.231+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:50.670 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:50.232+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:50.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:50.282+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:50.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:50.282+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:50.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:50.282+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:50.670 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:50.282+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:50.924 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:50.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:50.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:50.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:50.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:50.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:50.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:50.666+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:51.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:51.267+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:51.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:51.267+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:51.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:51.267+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:51.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:51.267+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:51.573 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:51.187+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:51.573 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:51.187+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:51.573 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:51.187+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:51.574 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:51.188+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:51.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:51.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:51.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:51.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:51.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:51.627+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:51.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:51.628+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:52.150+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:52.150+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:52.150+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:52.150+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:52.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:52.255+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:52.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:52.255+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:52.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:52.255+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:52.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:52.255+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:52.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:52.663+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:52.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:52.663+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:52.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:52.663+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:52.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:52.663+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:53.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:53.223+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:53.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:53.223+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:53.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:53.223+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:53.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:53.224+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:53.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:53.118+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:53.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:53.118+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:53.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:53.118+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:53.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:53.118+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:54.079 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:53.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:54.079 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:53.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:54.079 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:53.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:54.079 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:53.705+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:54.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:43:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:43:54.074+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23902 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:43:54.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:54.218+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:54.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:54.218+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:54.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:54.218+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:54.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:54.219+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:54.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:54.124+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:54.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:54.124+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:54.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:54.124+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:54.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:54.124+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:55.137 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:54.680+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:55.137 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:54.680+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:55.137 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:54.680+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:55.137 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:54.680+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:55.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:55.134+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:55.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:55.134+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:55.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:55.134+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:55.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:55.134+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:55.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:55.173+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:55.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:55.173+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:55.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:55.173+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:55.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:55.174+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:55.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:55.694+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:55.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:55.696+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:55.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:55.696+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:55.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:55.696+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:56.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:43:56.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:43:56.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:56.110+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:56.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:56.110+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:56.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:56.110+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:56.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:56.111+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:56.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:56.201+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:56.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:56.201+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:56.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:56.201+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:56.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:56.202+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:57.159 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:56.681+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:57.159 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:56.681+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:57.159 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:56.681+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:57.160 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:56.682+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:57.156+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:57.156+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:57.156+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:57.157+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:57.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:57.181+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:57.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:57.181+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:57.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:57.181+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:57.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:57.181+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:58.157 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:57.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:58.157 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:57.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:58.157 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:57.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:58.157 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:57.712+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:58.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:58.195+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:58.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:58.195+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:58.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:58.195+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:58.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:58.196+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:58.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:58.154+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:58.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:58.154+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:58.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:58.154+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:58.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:58.155+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:59.081 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:58.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:59.081 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:58.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:59.081 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:58.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:43:59.081 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:58.702+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:43:59.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:43:59.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:43:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:43:59.078+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 23955 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:43:59.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:59.155+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:59.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:59.155+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:59.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:59.155+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:43:59.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:43:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:43:59.156+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:43:59.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:59.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:59.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:59.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:59.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:59.176+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:43:59.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:43:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:43:59.176+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:00.150 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:59.681+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:00.150 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:59.681+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:00.150 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:59.681+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:00.150 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:43:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:43:59.681+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:00.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:00.189+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:00.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:00.189+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:00.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:00.189+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:00.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:00.190+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:00.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:00.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:00.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:00.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:00.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:00.147+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:00.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:00.148+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:01.140 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:00.682+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:01.140 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:00.682+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:01.140 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:00.682+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:01.140 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:00.683+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:01.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:01.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:01.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:01.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:01.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:01.162+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:01.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:01.163+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:01.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:01.137+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:01.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:01.137+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:01.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:01.137+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:01.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:01.138+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:01.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:01.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:01.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:01.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:01.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:01.700+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:01.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:01.700+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:02.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:44:02.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:44:02.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:02.123+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:02.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:02.123+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:02.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:02.123+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:02.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:02.124+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:02.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:02.102+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:02.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:02.102+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:02.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:02.102+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:02.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:02.104+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:02.430 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:44:02.430 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:44:02.431 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-27T04:44:02.431 DEBUG:teuthology.orchestra.run.smithi046:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-27T04:44:02.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:44:02.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:44:02.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:02.667+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:02.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:02.667+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:02.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:02.667+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:02.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:02.669+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:03.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:03.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:03.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:03.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:03.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:03.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:03.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:03.069+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:03.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:03.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:03.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:03.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:03.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:03.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:03.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:03.086+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:03.635+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:03.635+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:03.635+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:03.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:03.636+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:04.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:44:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:44:04.080+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24007 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:44:04.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:04.132+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:04.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:04.132+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:04.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:04.132+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:04.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:04.132+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:04.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:04.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:04.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:04.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:04.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:04.040+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:04.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:04.040+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:04.887 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:04.603+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:04.888 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:04.603+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:04.888 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:04.603+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:04.888 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:04.604+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:05.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:05.132+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:05.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:05.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:05.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:05.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:05.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:05.133+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:05.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:05.060+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:05.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:05.060+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:05.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:05.060+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:05.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:05.061+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:05.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:44:05.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:05.577+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:05.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:05.577+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:05.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:05.577+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:05.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:05.578+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:06.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:06.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:06.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:06.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:06.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:06.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:06.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:06.090+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:06.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:06.056+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:06.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:06.056+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:06.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:06.056+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:06.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:06.057+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:06.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:06.599+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:06.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:06.599+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:06.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:06.599+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:06.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:06.600+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:07.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:07.036+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:07.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:07.036+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:07.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:07.036+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:07.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:07.037+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:07.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:07.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:07.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:07.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:07.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:07.086+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:07.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:07.087+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:07.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:07.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:07.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:07.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:07.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:07.648+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:07.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:07.649+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:08.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:08.028+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:08.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:08.028+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:08.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:08.028+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:08.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:08.029+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:08.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:08.093+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:08.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:08.093+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:08.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:08.093+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:08.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:08.094+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:08.997 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:08.677+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:08.997 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:08.677+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:08.997 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:08.677+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:08.998 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:08.678+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:09.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:44:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:44:09.083+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24060 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:44:09.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:09.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:09.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:09.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:09.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:09.081+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:09.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:09.082+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:09.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:08.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:09.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:08.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:09.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:08.994+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:09.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:08.995+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:09.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:09.672+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:09.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:09.672+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:09.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:09.672+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:09.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:09.673+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:10.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:10.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:10.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:10.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:10.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:10.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:10.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:10.053+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:10.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:09.996+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:10.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:09.996+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:10.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:09.996+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:10.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:09.997+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:11.002 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:10.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:11.002 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:10.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:11.003 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:10.717+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:11.003 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:10.718+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:11.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:11.026+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:11.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:11.026+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:11.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:11.026+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:11.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:11.027+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:11.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:10.999+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:11.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:10.999+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:11.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:10.999+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:11.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:11.001+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:11.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:11.691+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:11.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:11.691+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:11.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:11.691+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:11.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:11.692+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:12.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:12.045+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:12.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:12.045+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:12.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:12.045+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:12.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:12.045+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:12.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:12.026+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:12.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:12.026+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:12.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:12.026+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:12.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:12.026+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:13.034 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:12.725+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:13.035 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:12.725+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:13.035 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:12.725+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:13.035 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:12.725+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:13.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:13.067+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:13.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:13.067+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:13.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:13.067+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:13.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:13.067+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:13.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:13.032+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:13.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:13.032+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:13.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:13.032+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:13.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:13.033+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:14.015 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:13.749+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:14.015 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:13.749+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:14.015 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:13.749+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:14.015 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:13.750+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:14.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:44:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:44:14.085+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24112 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:44:14.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:14.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:14.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:14.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:14.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:14.012+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:14.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:14.013+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:14.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:14.087+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:14.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:14.087+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:14.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:14.087+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:14.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:14.088+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:14.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:44:14.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:44:15.034 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:14.751+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:15.035 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:14.751+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:15.035 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:14.752+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:15.035 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:14.753+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:15.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:15.032+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:15.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:15.032+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:15.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:15.032+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:15.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:15.033+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:15.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:15.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:15.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:15.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:15.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:15.133+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:15.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:15.133+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:16.084 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:15.719+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:16.085 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:15.719+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:16.085 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:15.719+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:16.085 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:15.720+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:16.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:16.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:16.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:16.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:16.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:16.082+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:16.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:16.083+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:16.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:16.123+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:16.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:16.123+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:16.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:16.123+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:16.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:16.124+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:17.098 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:16.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:17.099 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:16.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:17.099 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:16.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:17.099 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:16.735+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:17.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:17.096+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:17.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:17.096+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:17.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:17.096+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:17.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:17.097+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:17.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:17.108+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:17.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:17.108+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:17.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:17.108+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:17.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:17.108+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:17.566 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:44:18.077 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:17.701+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:18.077 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:17.701+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:18.077 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:17.701+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:18.077 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:17.702+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:18.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:18.074+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:18.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:18.074+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:18.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:18.074+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:18.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:18.075+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:18.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:18.092+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:18.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:18.092+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:18.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:18.092+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:18.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:18.093+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:18.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:18.654+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:18.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:18.654+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:18.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:18.654+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:18.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:18.655+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:19.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:44:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:44:19.087+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24165 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:44:19.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:19.037+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:19.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:19.037+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:19.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:19.037+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:19.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:19.039+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:19.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:19.118+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:19.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:19.118+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:19.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:19.118+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:19.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:19.118+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:19.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:19.637+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:19.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:19.637+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:19.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:19.637+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:19.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:19.637+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:20.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:20.001+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:20.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:20.001+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:20.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:20.001+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:20.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:20.002+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:20.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:20.093+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:20.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:20.093+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:20.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:20.093+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:20.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:20.093+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:20.966 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:20.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:20.966 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:20.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:20.966 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:20.686+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:20.966 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:20.687+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:21.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:21.075+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:21.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:21.075+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:21.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:21.075+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:21.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:21.076+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:21.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:20.963+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:21.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:20.963+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:21.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:20.963+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:21.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:20.964+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:21.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:21.657+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:21.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:21.657+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:21.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:21.657+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:21.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:21.658+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:22.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:22.032+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:22.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:22.032+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:22.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:22.032+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:22.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:22.032+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:22.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:21.980+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:22.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:21.980+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:22.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:21.980+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:22.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:21.981+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:22.975 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:22.696+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:22.975 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:22.696+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:22.975 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:22.696+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:22.976 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:22.697+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:23.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:22.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:23.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:22.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:23.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:22.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:23.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:22.974+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:23.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:23.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:23.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:23.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:23.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:23.036+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:23.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:23.037+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:23.995 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:23.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:23.995 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:23.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:23.996 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:23.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:23.996 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:23.679+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:24.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:44:24.090+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24217 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:44:24.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:23.999+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:24.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:23.999+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:24.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:23.999+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:24.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:24.001+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:24.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:23.993+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:24.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:23.993+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:24.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:23.993+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:24.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:23.993+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:24.977 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:24.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:24.977 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:24.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:24.977 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:24.721+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:24.977 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:24.722+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:25.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:24.974+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:25.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:24.974+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:25.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:24.974+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:25.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:24.975+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:25.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:24.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:25.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:24.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:25.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:24.997+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:25.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:24.997+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:25.976 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:25.719+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:25.976 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:25.719+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:25.976 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:25.719+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:25.976 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:25.720+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:26.391 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:25.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:26.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:25.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:26.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:25.973+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:26.392 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:25.974+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:26.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:26.027+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:26.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:26.027+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:26.392 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:26.027+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:26.393 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:26.028+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:27.014 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:26.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:27.014 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:26.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:27.014 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:26.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:27.014 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:26.679+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:27.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:27.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:27.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:27.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:27.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:27.052+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:27.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:27.052+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:27.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:27.011+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:27.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:27.011+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:27.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:27.011+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:27.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:27.012+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:27.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:27.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:27.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:27.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:27.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:27.666+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:27.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:27.666+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:28.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:28.013+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:28.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:28.013+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:28.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:28.013+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:28.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:28.014+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:28.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:28.020+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:28.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:28.020+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:28.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:28.020+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:28.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:28.021+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:28.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:28.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:28.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:28.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:28.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:28.653+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:28.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:28.654+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:29.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:44:29.093+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24270 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:44:29.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:29.021+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:29.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:29.021+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:29.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:29.021+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:29.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:29.021+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:29.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:29.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:29.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:29.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:29.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:29.058+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:29.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:29.060+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:29.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:44:29.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:44:29.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:29.614+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:29.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:29.614+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:29.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:29.614+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:29.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:29.615+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:30.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:30.020+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:30.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:30.020+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:30.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:30.020+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:30.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:30.020+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:30.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:29.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:30.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:29.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:30.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:29.978+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:30.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:29.978+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:30.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:30.611+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:30.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:30.611+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:30.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:30.611+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:30.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:30.612+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:31.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:31.011+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:31.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:31.011+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:31.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:31.011+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:31.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:31.012+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:31.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:30.992+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:31.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:30.992+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:31.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:30.992+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:31.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:30.992+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:31.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:31.634+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:31.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:31.634+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:31.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:31.634+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:31.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:31.635+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:32.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:32.026+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:32.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:32.026+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:32.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:32.026+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:32.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:32.027+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:32.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:32.014+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:32.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:32.014+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:32.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:32.014+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:32.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:32.015+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:32.638 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:44:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:32.616+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:32.616+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:32.616+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:32.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:32.617+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:33.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:33.016+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:33.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:33.016+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:33.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:33.016+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:33.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:33.017+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:33.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:33.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:33.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:33.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:33.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:33.022+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:33.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:33.023+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:33.588+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:33.588+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:33.588+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:33.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:33.589+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:34.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:44:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:44:34.095+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24322 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:44:34.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:34.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:34.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:34.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:34.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:34.064+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:34.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:34.065+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:34.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:34.065+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:34.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:34.065+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:34.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:34.065+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:34.427 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:34.066+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:34.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:34.605+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:34.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:34.605+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:34.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:34.605+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:34.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:34.606+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:35.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:35.034+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:35.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:35.034+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:35.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:35.034+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:35.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:35.035+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:35.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:35.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:35.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:35.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:35.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:35.089+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:35.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:35.090+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:35.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:44:35.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:44:35.710 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:44:35.710 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:44:35.711 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-27T04:44:35.712 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-27_03:08:09-upgrade-main-distro-default-smithi/7675777/remote/smithi046/crash 2024-04-27T04:44:35.713 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/crash -- . 2024-04-27T04:44:35.775 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/lib/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-27_03:08:09-upgrade-main-distro-default-smithi/7675777/remote/smithi073/crash 2024-04-27T04:44:35.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:44:35.777 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:44:35.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:35.634+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:35.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:35.634+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:35.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:35.634+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:35.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:35.635+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:36.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:36.038+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:36.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:36.038+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:36.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:36.038+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:36.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:36.039+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:36.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:36.084+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:36.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:36.084+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:36.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:36.084+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:36.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:36.084+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:36.662+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:36.662+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:36.662+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:36.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:36.663+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:37.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:37.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:37.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:37.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:37.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:37.071+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:37.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:37.072+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:37.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:37.069+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:37.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:37.069+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:37.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:37.069+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:37.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:37.070+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:37.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:37.640+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:37.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:37.640+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:37.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:37.640+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:37.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:37.641+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:38.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:38.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:38.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:38.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:38.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:38.067+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:38.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:38.068+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:38.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:38.113+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:38.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:38.113+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:38.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:38.113+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:38.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:38.114+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:38.846 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:44:38.920 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:38.609+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:38.920 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:38.609+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:38.920 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:38.609+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:38.920 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:38.610+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:39.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:44:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:44:39.097+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24374 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:44:39.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:39.120+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:39.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:39.120+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:39.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:39.120+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:39.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:39.120+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:39.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:39.035+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:39.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:39.035+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:39.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:39.035+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:39.176 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:39.036+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:39.909 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:39.633+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:39.909 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:39.633+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:39.910 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:39.633+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:39.910 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:39.633+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:40.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:40.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:40.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:40.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:40.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:40.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:40.175 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:40.047+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:40.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:40.132+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:40.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:40.132+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:40.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:40.132+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:40.176 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:40.132+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:40.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:40.637+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:40.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:40.637+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:40.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:40.637+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:40.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:40.638+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:41.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:41.069+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:41.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:41.069+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:41.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:41.069+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:41.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:41.070+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:41.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:41.170+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:41.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:41.170+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:41.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:41.170+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:41.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:41.171+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:41.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:41.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:41.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:41.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:41.679+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:42.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:42.048+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:42.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:42.048+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:42.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:42.048+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:42.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:42.049+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:42.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:42.216+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:42.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:42.216+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:42.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:42.216+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:42.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:42.216+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:43.058 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:42.727+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:43.059 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:42.727+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:43.059 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:42.727+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:43.059 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:42.728+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:43.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:43.056+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:43.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:43.056+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:43.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:43.056+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:43.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:43.057+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:43.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:43.243+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:43.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:43.243+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:43.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:43.243+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:43.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:43.244+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:44.049 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:43.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:44.049 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:43.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:44.049 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:43.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:44.050 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:43.704+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:44.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:44:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:44:44.100+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24425 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:44:44.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:44.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:44.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:44.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:44.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:44.046+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:44.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:44.047+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:44.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:44.231+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:44.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:44.231+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:44.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:44.231+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:44.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:44.231+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:45.079 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:44.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:45.079 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:44.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:45.079 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:44.704+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:45.079 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:44.705+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:45.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:45.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:45.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:45.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:45.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:45.076+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:45.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:45.077+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:45.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:45.261+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:45.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:45.261+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:45.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:45.261+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:45.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:45.263+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:46.043 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:45.701+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:46.043 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:45.701+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:46.043 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:45.701+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:46.044 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:45.701+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:46.041+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:46.041+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:46.041+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:46.042+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:46.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:46.276+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:46.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:46.276+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:46.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:46.276+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:46.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:46.277+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:47.075 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:46.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:47.075 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:46.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:47.075 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:46.718+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:47.075 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:46.719+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:47.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:47.263+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:47.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:47.263+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:47.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:47.263+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:47.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:47.263+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:47.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:47.072+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:47.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:47.072+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:47.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:47.072+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:47.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:47.073+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:47.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:44:47.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:44:48.094 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:47.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:48.095 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:47.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:48.095 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:47.712+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:48.095 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:47.713+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:48.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:48.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:48.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:48.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:48.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:48.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:48.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:48.093+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:48.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:48.256+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:48.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:48.256+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:48.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:48.256+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:48.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:48.257+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:49.090 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:48.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:49.090 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:48.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:49.091 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:48.703+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:49.091 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:48.704+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:49.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:44:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:44:49.103+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24478 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:44:49.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:49.087+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:49.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:49.087+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:49.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:49.087+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:49.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:49.088+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:49.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:49.276+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:49.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:49.276+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:49.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:49.276+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:49.427 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:49.277+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:49.670+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:49.670+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:49.670+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:49.671+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:50.264 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:50.134+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:50.264 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:50.134+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:50.264 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:50.134+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:50.265 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:50.135+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:50.627 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:50.261+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:50.627 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:50.261+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:50.627 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:50.261+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:50.628 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:50.263+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:50.904 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:50.651+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:50.904 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:50.651+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:50.904 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:50.651+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:50.904 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:50.652+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:50.910 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:44:51.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:51.251+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:51.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:51.251+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:51.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:51.251+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:51.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:51.251+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:51.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:51.155+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:51.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:51.155+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:51.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:51.155+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:51.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:51.156+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:52.142 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:51.696+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:52.142 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:51.696+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:52.142 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:51.696+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:52.142 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:51.697+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:52.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:52.205+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:52.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:52.205+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:52.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:52.205+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:52.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:52.206+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:52.139+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:52.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:52.139+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:52.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:52.139+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:52.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:52.140+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:53.137 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:52.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:53.137 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:52.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:53.137 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:52.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:53.137 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:52.734+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:53.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:53.160+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:53.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:53.160+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:53.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:53.160+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:53.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:53.160+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:53.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:53.140+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:53.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:53.140+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:53.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:53.140+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:53.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:53.141+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:54.040 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:53.711+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:54.040 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:53.711+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:54.040 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:53.711+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:54.040 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:53.712+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:54.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:44:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:44:54.105+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24530 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:44:54.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:54.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:54.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:54.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:54.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:54.092+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:54.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:54.093+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:54.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:54.153+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:54.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:54.153+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:54.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:54.153+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:54.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:54.154+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:55.131 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:54.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:55.131 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:54.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:55.131 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:54.678+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:55.132 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:54.678+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:55.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:55.128+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:55.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:55.128+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:55.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:55.128+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:55.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:55.130+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:55.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:55.200+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:55.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:55.200+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:55.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:55.200+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:55.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:55.200+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:56.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:55.683+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:56.170 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:55.683+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:56.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:55.683+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:56.171 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:55.683+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:56.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:56.239+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:56.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:56.239+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:56.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:56.239+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:56.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:56.240+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:56.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:56.168+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:56.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:56.168+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:56.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:56.168+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:56.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:56.169+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:57.135 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:56.701+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:57.135 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:56.701+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:57.136 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:56.701+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:57.136 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:56.702+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:57.133+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:57.133+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:57.133+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:57.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:57.134+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:57.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:57.258+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:57.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:57.258+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:57.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:57.258+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:57.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:57.258+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:58.116 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:57.713+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:58.116 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:57.713+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:58.116 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:57.713+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:58.116 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:57.714+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:58.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:58.113+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:58.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:58.113+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:58.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:58.113+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:58.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:58.114+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:58.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:58.267+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:58.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:58.267+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:58.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:58.267+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:58.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:58.268+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:59.110 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:58.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:59.111 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:58.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:59.111 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:58.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:44:59.111 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:58.716+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:44:59.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:44:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:44:59.108+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24583 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:44:59.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:59.116+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:59.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:59.116+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:59.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:59.116+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:44:59.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:44:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:44:59.117+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:44:59.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:59.286+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:59.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:59.286+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:59.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:59.286+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:44:59.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:44:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:44:59.287+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:00.157 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:59.676+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:00.158 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:59.676+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:00.158 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:59.676+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:00.158 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:44:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:44:59.677+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:00.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:00.155+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:00.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:00.155+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:00.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:00.155+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:00.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:00.155+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:00.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:00.305+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:00.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:00.305+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:00.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:00.305+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:00.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:00.305+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:00.639+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:00.639+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:00.639+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:00.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:00.640+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:01.571 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:01.196+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:01.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:01.196+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:01.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:01.196+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:01.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:01.197+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:01.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:01.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:01.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:01.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:01.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:01.272+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:01.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:01.273+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:01.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:01.660+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:01.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:01.660+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:01.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:01.660+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:01.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:01.661+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:02.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:02.234+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:02.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:02.234+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:02.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:02.234+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:02.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:02.235+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:02.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:02.290+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:02.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:02.290+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:02.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:02.290+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:02.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:02.291+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:02.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:45:02.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:45:03.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:02.709+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:03.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:02.709+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:03.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:02.709+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:03.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:02.710+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:03.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:03.279+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:03.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:03.279+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:03.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:03.279+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:03.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:03.280+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:03.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:03.284+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:03.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:03.284+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:03.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:03.284+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:03.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:03.284+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:04.113 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:03.740+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:04.114 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:03.740+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:04.114 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:03.741+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:04.114 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:03.742+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:04.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:45:04.110+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24635 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:45:04.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:04.246+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:04.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:04.246+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:04.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:04.246+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:04.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:04.247+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:04.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:04.244+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:04.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:04.244+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:04.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:04.244+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:04.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:04.245+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:05.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:04.741+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:05.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:04.741+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:05.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:04.741+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:05.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:04.742+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:05.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:05.231+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:05.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:05.231+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:05.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:05.231+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:05.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:05.232+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:05.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:05.264+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:05.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:05.264+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:05.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:05.264+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:05.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:05.265+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:05.982 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:45:06.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:05.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:06.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:05.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:06.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:05.748+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:06.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:05.749+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:06.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:06.227+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:06.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:06.227+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:06.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:06.227+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:06.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:06.228+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:06.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:06.234+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:06.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:06.234+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:06.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:06.234+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:06.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:06.235+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:07.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:06.783+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:07.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:06.783+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:07.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:06.783+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:07.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:06.784+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:07.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:07.216+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:07.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:07.216+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:07.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:07.216+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:07.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:07.217+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:07.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:07.207+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:07.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:07.207+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:07.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:07.207+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:07.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:07.208+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:08.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:07.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:08.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:07.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:08.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:07.734+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:08.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:07.735+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:08.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:08.210+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:08.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:08.210+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:08.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:08.210+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:08.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:08.211+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:08.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:08.181+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:08.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:08.181+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:08.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:08.181+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:08.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:08.182+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:08.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:45:08.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:45:09.054 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:45:09.054 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:45:09.055 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-27T04:45:09.055 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-27T04:45:09.085 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:08.688+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:09.085 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:08.688+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:09.085 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:08.688+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:09.085 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:08.689+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:09.092 INFO:tasks.cephadm:Compressing logs... 2024-04-27T04:45:09.093 DEBUG:teuthology.orchestra.run.smithi046:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-27T04:45:09.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:45:09.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:45:09.157 INFO:teuthology.orchestra.run.smithi046.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory 2024-04-27T04:45:09.157 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-mon.smithi046.log 2024-04-27T04:45:09.159 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph.log 2024-04-27T04:45:09.159 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-27T04:45:09.160 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-mon.smithi046.log: /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-27T04:45:09.160 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi046.log 2024-04-27T04:45:09.160 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-mgr.smithi046.qrfdvr.log 2024-04-27T04:45:09.160 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi046.log: 86.9%/var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph.audit.log: -- replaced with /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-27T04:45:09.160 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-27T04:45:09.163 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose/var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-mgr.smithi046.qrfdvr.log: -- /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-27T04:45:09.163 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi046.mycakd.log 2024-04-27T04:45:09.163 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-osd.0.log: 90.6% -- replaced with /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-27T04:45:09.164 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-27T04:45:09.164 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi046.mycakd.log: 94.5% -- replaced with /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi046.log.gz 2024-04-27T04:45:09.164 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-27T04:45:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.5% -- replaced with /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-27T04:45:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi046.qgmaql.log 2024-04-27T04:45:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-27T04:45:09.187 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi046.qgmaql.log: /var/log/ceph/cephadm.log: 91.0% -- replaced with /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-27T04:45:09.201 INFO:teuthology.orchestra.run.smithi046.stderr: 90.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-27T04:45:09.388 INFO:teuthology.orchestra.run.smithi046.stderr: 90.9% -- replaced with /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-mgr.smithi046.qrfdvr.log.gz 2024-04-27T04:45:09.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:09.156+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:09.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:09.156+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:09.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:09.156+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:09.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:09.158+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:09.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:09.238+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:09.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:09.238+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:09.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:09.238+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:09.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:09.239+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:09.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:45:09.112+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24688 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:45:10.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:09.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:10.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:09.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:10.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:09.716+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:10.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:09.717+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:10.319 INFO:teuthology.orchestra.run.smithi046.stderr:gzip: /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-mon.smithi046.log: file size changed while zipping 2024-04-27T04:45:10.319 INFO:teuthology.orchestra.run.smithi046.stderr: 92.8% -- replaced with /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-mon.smithi046.log.gz 2024-04-27T04:45:10.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:10.198+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:10.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:10.198+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:10.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:10.198+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:10.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:10.199+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:10.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:10.197+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:10.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:10.197+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:10.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:10.197+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:10.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:10.198+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:11.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:10.726+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:11.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:10.726+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:11.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:10.726+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:11.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:10.727+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:11.573 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:11.193+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:11.573 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:11.193+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:11.573 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:11.193+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:11.574 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:11.194+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:11.574 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:11.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:11.574 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:11.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:11.574 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:11.217+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:11.575 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:11.218+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:11.587 INFO:teuthology.orchestra.run.smithi046.stderr: 95.2% -- replaced with /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi046.qgmaql.log.gz 2024-04-27T04:45:11.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:11.723+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:11.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:11.723+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:11.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:11.723+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:11.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:11.724+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:12.190 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:45:12.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:12.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:12.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:12.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:12.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:12.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:12.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:12.189+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:12.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:12.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:12.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:12.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:12.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:12.198+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:12.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:12.199+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:12.902 INFO:teuthology.orchestra.run.smithi046.stderr:gzip: /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-osd.2.log: file size changed while zipping 2024-04-27T04:45:12.902 INFO:teuthology.orchestra.run.smithi046.stderr: 91.2% -- replaced with /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-27T04:45:13.165 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:12.772+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:13.165 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:12.772+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:13.165 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:12.772+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:13.166 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:12.773+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:13.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:13.223+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:13.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:13.223+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:13.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:13.223+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:13.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:13.224+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:13.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:13.162+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:13.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:13.162+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:13.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:13.162+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:13.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:13.163+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:14.120 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:13.823+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:14.120 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:13.823+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:14.121 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:13.823+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:14.121 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:13.824+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:14.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:45:14.115+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24740 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:45:14.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:14.257+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:14.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:14.257+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:14.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:14.257+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:14.426 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:14.258+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:14.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:14.136+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:14.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:14.136+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:14.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:14.136+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:14.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:14.137+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:15.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:14.857+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:15.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:14.857+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:15.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:14.857+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:15.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:14.858+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:gzip: /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-osd.0.log: file size changed while zipping 2024-04-27T04:45:15.410 INFO:teuthology.orchestra.run.smithi046.stderr: 94.9% -- replaced with /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-27T04:45:15.607 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:15.177+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:15.608 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:15.177+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:15.608 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:15.177+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:15.608 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:15.178+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:15.608 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:15.236+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:15.608 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:15.236+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:15.609 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:15.236+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:15.609 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:15.237+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:15.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:15.809+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:15.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:15.809+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:15.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:15.809+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:15.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:15.810+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:16.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:16.211+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:16.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:16.211+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:16.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:16.211+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:16.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:16.212+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:16.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:16.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:16.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:16.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:16.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:16.274+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:16.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:16.275+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:17.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:16.860+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:17.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:16.860+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:17.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:16.860+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:17.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:16 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:16.860+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:17.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:17.229+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:17.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:17.229+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:17.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:17.229+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:17.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:17.230+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:17.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:17.314+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:17.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:17.314+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:17.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:17.314+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:17.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:17.315+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:18.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:17.861+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:18.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:17.861+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:18.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:17.861+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:18.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:17 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:17.862+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:18.540 INFO:teuthology.orchestra.run.smithi046.stderr:gzip: /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-osd.1.log: file size changed while zipping 2024-04-27T04:45:18.541 INFO:teuthology.orchestra.run.smithi046.stderr: 96.3% -- replaced with /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-27T04:45:18.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:18.212+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:18.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:18.212+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:18.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:18.212+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:18.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:18.213+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:18.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:18.334+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:18.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:18.334+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:18.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:18.334+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:18.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:18.335+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:19.120 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:18.820+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:19.120 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:18.820+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:19.121 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:18.820+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:19.121 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:18 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:18.821+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:19.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:45:19.116+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24791 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:45:19.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:19.303+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:19.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:19.303+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:19.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:19.303+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:19.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:19.303+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:19.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:19.169+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:19.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:19.169+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:19.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:19.169+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:19.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:19.169+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:20.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:19.835+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:20.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:19.835+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:20.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:19.835+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:20.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:19 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:19.836+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:20.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:20.218+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:20.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:20.218+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:20.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:20.218+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:20.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:20.219+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:20.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:20.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:20.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:20.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:20.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:20.294+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:20.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:20.295+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:21.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:20.878+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:21.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:20.878+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:21.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:20.878+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:21.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:20 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:20.879+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:21.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:45:21.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:45:21.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:21.326+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:21.572 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:21.326+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:21.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:21.326+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:21.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:21.327+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:21.573 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:21.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:21.573 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:21.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:21.574 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:21.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:21.574 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:21.259+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:22.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:21.905+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:22.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:21.905+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:22.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:21.905+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:22.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:21 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:21.907+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:22.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:22.229+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:22.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:22.229+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:22.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:22.229+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:22.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:22.231+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:22.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:22.300+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:22.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:22.300+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:22.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:22.300+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:22.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:22.300+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:23.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:22.897+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:23.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:22.897+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:23.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:22.897+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:23.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:22 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:22.898+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:23.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:23.292+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:23.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:23.292+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:23.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:23.292+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:23.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:23.293+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:23.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:23.224+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:23.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:23.224+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:23.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:23.224+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:23.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:23.226+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:24.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:45:24.262 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:45:24.119+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24843 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:45:24.262 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:23.933+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:24.262 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:23.933+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:24.263 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:23.933+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:24.263 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:23 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:23.933+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:24.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:24.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:24.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:24.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:24.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:24.258+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:24.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:24.259+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:24.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:24.331+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:24.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:24.331+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:24.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:24.331+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:24.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:24.332+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:25.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:24.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:25.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:24.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:25.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:24.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:25.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:24 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:24.979+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:25.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:25.371+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:25.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:25.371+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:25.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:25.371+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:25.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:25.371+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:25.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:25.301+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:25.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:25.301+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:25.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:25.301+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:25.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:25.302+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:26.281 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:25.972+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:26.281 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:25.972+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:26.281 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:25.972+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:26.281 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:25 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:25.973+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:26.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:26.277+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:26.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:26.277+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:26.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:26.277+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:26.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:26.278+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:26.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:26.398+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:26.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:26.398+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:26.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:26.398+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:26.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:26.398+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:27.255 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:26.944+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:27.255 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:26.944+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:27.255 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:26.944+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:27.255 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:26 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:26.944+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:27.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:27.350+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:27.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:27.350+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:27.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:27.350+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:27.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:27.351+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:27.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:27.251+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:27.677 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:27.251+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:27.677 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:27.251+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:27.677 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:27.253+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:28.284 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:27.957+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:28.285 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:27.957+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:28.285 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:27.957+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:28.285 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:27 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:27.958+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:28.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:28.281+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:28.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:28.281+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:28.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:28.281+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:28.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:28.282+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:28.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:28.357+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:28.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:28.357+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:28.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:28.357+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:28.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:28.359+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:29.284 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:45:29.121+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24896 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:45:29.285 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:28.995+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:29.285 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:28.995+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:29.285 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:28.995+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:29.285 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:28 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:28.995+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:29.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:29.281+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:29.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:29.281+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:29.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:29.281+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:29.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:29.281+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:29.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:29.355+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:29.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:29.355+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:29.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:29.355+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:29.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:29 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:29.356+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:30.294 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:30.011+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:30.295 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:30.011+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:30.295 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:30.011+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:30.295 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:30.012+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:30.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:30.363+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:30.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:30.363+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:30.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:30.363+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:30.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:30.363+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:30.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:30.291+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:30.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:30.291+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:30.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:30.291+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:30.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:30 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:30.292+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:31.304 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:30.999+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:31.304 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:30.999+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:31.304 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:30.999+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:31.305 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:31.000+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:31.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:31.300+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:31.572 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:31.300+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:31.573 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:31.300+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:31.573 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:31.302+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:31.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:31.402+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:31.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:31.402+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:31.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:31.402+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:31.573 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:31.403+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:32.016 INFO:teuthology.orchestra.run.smithi046.stderr: 91.6% -- replaced with /var/log/ceph/b1d3a2d0-0449-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi046.mycakd.log.gz 2024-04-27T04:45:32.018 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-04-27T04:45:32.019 INFO:teuthology.orchestra.run.smithi046.stderr:real 0m22.871s 2024-04-27T04:45:32.019 INFO:teuthology.orchestra.run.smithi046.stderr:user 0m44.763s 2024-04-27T04:45:32.019 INFO:teuthology.orchestra.run.smithi046.stderr:sys 0m1.132s 2024-04-27T04:45:32.341 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:31.956+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:32.341 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:31.956+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:32.341 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:31.956+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:32.341 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:31 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:31.956+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:32.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:32.338+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:32.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:32.338+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:32.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:32.338+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:32.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:32.339+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:32.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:32.429+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:32.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:32.429+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:32.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:32.429+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:32.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:32.430+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:33.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:32.913+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:33.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:32.913+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:33.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:32.913+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:33.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:32 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:32.914+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:33.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:33.420+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:33.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:33.420+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:33.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:33.420+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:33.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:33.422+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:33.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:33.302+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:33.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:33.302+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:33.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:33.302+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:33.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:33.302+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:34.289 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:45:34.125+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24947 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:45:34.289 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:33.933+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:34.289 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:33.933+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:34.290 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:33.933+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:34.290 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:33 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:33.934+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:34.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:34.444+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:34.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:34.444+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:34.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:34.444+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:34.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:34.445+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:34.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:34.287+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:34.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:34.287+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:34.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:34.287+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:34.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:34.287+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:35.243 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:34.980+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:35.243 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:34.980+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:35.243 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:34.980+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:35.243 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:34 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:34.981+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:35.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:35.444+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:35.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:35.444+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:35.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:35.444+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:35.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:35.444+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:35.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:35.240+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:35.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:35.240+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:35.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:35.240+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:35.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:35 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:35.241+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:36.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:45:36.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:45:36.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:36.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:36.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:36.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:36.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:36.000+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:36.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:36.001+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:36.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:36.236+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:36.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:36.236+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:36.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:36.236+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:36.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:36.237+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:36.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:36.453+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:36.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:36.453+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:36.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:36.453+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:36.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:36.453+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:37.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:36.990+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:37.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:36.990+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:37.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:36.990+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:37.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:36 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:36.991+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:37.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:37.191+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:37.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:37.191+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:37.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:37.191+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:37.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:37.192+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:37.717 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:37.435+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:37.718 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:37.435+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:37.718 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:37.435+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:37.718 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:37.435+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:38.407 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:38.152+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:38.408 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:38.152+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:38.408 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:38.152+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:38.408 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:38.153+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:38.408 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:37.968+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:38.408 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:37.968+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:38.408 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:37.968+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:38.409 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:37 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:37.969+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:38.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:38.405+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:38.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:38.405+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:38.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:38.405+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:38.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:38.405+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:39.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:45:39.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:45:39.127+0000 7f50b5721700 -1 mon.smithi046@0(probing) e2 get_health_metrics reporting 24999 slow ops, oldest is mgrbeacon mgr.smithi046.qrfdvr(b1d3a2d0-0449-11ef-bc93-c7b262605968,14664, , 0) 2024-04-27T04:45:39.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:39.200+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:39.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:39.200+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:39.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:39.200+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:39.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:39.201+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:39.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:38.963+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:39.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:38.963+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:39.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:38.963+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:39.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:38 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:38.964+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:39.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:39.447+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:39.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:39.447+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:39.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:39.447+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:39.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:39.447+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:40.206 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:39.945+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:40.206 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:39.945+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:40.206 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:39.945+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:40.207 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:39 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:39.946+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:40.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:40.434+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:40.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:40.434+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:40.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:40.434+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:40.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:40.434+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:40.677 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:40.203+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:40.677 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:40.203+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:40.677 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:40.203+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:40.677 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:40.204+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:41.415 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:41.162+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:41.415 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:41.162+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:41.415 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:41.162+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:41.415 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:41.162+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:41.415 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:40.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:41.416 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:40.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:41.416 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:40.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:41.416 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:40 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:40.978+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:41.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:41.412+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:41.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:41.412+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:41.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:41.412+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:41.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:41.413+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:42.191 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:41.928+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:42.191 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:41.928+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:42.191 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:41.928+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:42.192 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:41 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:41.930+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:42.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:45:42.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:45:42.398 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:45:42.398 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:45:42.399 INFO:tasks.cephadm:Removing cluster... 2024-04-27T04:45:42.399 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 --force 2024-04-27T04:45:42.565 INFO:teuthology.orchestra.run.smithi046.stdout:Deleting cluster with fsid: b1d3a2d0-0449-11ef-bc93-c7b262605968 2024-04-27T04:45:42.663 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:42.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:42.664 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:42.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:42.664 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:42.188+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:42.664 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:42.190+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:42.664 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:42.401+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:42.665 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:42.401+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:42.665 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:42.401+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:42.665 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:42.401+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:42.916 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:42 smithi046 systemd[1]: Stopping Ceph mon.smithi046 for b1d3a2d0-0449-11ef-bc93-c7b262605968... 2024-04-27T04:45:43.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:45:42.940+0000 7f50b8727700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi046 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T04:45:43.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046[26895]: 2024-04-27T04:45:42.940+0000 7f50b8727700 -1 mon.smithi046@0(probing) e2 *** Got Signal Terminated *** 2024-04-27T04:45:43.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:42.913+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:43.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:42.913+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:43.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:42.913+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:43.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:42 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:42.914+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:43.536 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:43 smithi046 podman[81397]: 2024-04-27 04:45:43.245512021 +0000 UTC m=+0.427624335 container died 72fac097effc4f80a0d126f7d48ff847c476190bc241aab5ea0cb6aaa8298132 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=reef-5403a02, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T04:45:43.537 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:43.197+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:43.537 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:43.197+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:43.537 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:43.197+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:43.537 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:43.198+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:43.537 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:43.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:43.538 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:43.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:43.538 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:43.407+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:43.538 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:43.408+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:43.895 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:43 smithi046 podman[81397]: 2024-04-27 04:45:43.642121091 +0000 UTC m=+0.824233422 container cleanup 72fac097effc4f80a0d126f7d48ff847c476190bc241aab5ea0cb6aaa8298132 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=reef-5403a02, ceph=True, org.label-schema.build-date=20240424) 2024-04-27T04:45:43.895 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:43 smithi046 bash[81397]: ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046 2024-04-27T04:45:44.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:43 smithi046 podman[81411]: 2024-04-27 04:45:43.89366903 +0000 UTC m=+0.642798977 container remove 72fac097effc4f80a0d126f7d48ff847c476190bc241aab5ea0cb6aaa8298132 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-mon-smithi046, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=reef-5403a02, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-27T04:45:44.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:44 smithi046 systemd[1]: ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@mon.smithi046.service: Deactivated successfully. 2024-04-27T04:45:44.175 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:44 smithi046 systemd[1]: Stopped Ceph mon.smithi046 for b1d3a2d0-0449-11ef-bc93-c7b262605968. 2024-04-27T04:45:44.176 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Apr 27 04:45:44 smithi046 systemd[1]: ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@mon.smithi046.service: Consumed 23.760s CPU time. 2024-04-27T04:45:44.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:43.955+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:44.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:43.955+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:44.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:43.955+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:44.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:43 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:43.956+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:44.529 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:44.204+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:44.530 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:44.204+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:44.530 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:44.204+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:44.530 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:44.205+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:44.530 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:44.451+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:44.531 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:44.451+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:44.531 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:44.451+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:44.531 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:44.452+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:45.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:44.920+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:45.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:44.920+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:45.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:44.920+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:45.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:44 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:44.921+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:45.465 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:45.237+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:45.465 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:45.237+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:45.465 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:45.237+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:45.465 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:45.238+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:45.849 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:45.462+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:45.850 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:45.462+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:45.850 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:45.462+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:45.850 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:45.463+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:46.156 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:45.935+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:46.156 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:45.935+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:46.156 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:45.935+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:46.156 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:45 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:45.936+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:46.275+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:46.275+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:46.275+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:46.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:46.276+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:46.850 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:46.438+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:46.850 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:46.438+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:46.851 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:46.438+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:46.851 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:46.438+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:47.119 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:46.924+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:47.119 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:46.924+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:47.119 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:46.924+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:47.119 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:46 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:46.925+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:47.403 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:47.232+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:47.404 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:47.232+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:47.404 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:47.232+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:47.404 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:47.233+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:47.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:47.449+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:47.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:47.449+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:47.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:47.449+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:47.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:47.450+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:48.156 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:47.881+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:48.156 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:47.881+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:48.157 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:47.881+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:48.157 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:47 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:47.882+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:48.407 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:48.200+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:48.407 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:48.200+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:48.407 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:48.200+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:48.407 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:48.201+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:48.408 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:48.403+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:48.408 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:48.403+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:48.408 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:48.403+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:48.408 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:48.404+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:49.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:48.869+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:49.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:48.869+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:49.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:48.869+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:49.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:48 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:48.870+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:49.657 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:49.225+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:49.658 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:49.225+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:49.658 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:49.225+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:49.658 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:49.226+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:49.658 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:49.426+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:49.658 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:49.426+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:49.658 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:49.426+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:49.659 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:49.427+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:49.884+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:49.884+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:49.884+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:49.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:49 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:49.885+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:50.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:50.251+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:50.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:50.251+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:50.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:50.251+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:50.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:50.251+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:50.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:50.459+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:50.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:50.459+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:50.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:50.459+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:50.677 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:50.460+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:50.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:50.895+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:50.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:50.895+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:50.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:50.895+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:50.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:50 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:50.896+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:51.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:51.264+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:51.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:51.264+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:51.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:51.264+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:51.677 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:51.265+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:51.677 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:51.440+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:51.678 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:51.440+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:51.678 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:51.440+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:51.678 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:51.441+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:52.106 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:51.913+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:52.106 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:51.913+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:52.106 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:51.913+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:52.106 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:51 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:51.914+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:52.288+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:52.288+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:52.288+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:52.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:52.289+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:52.692 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:52.469+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:52.693 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:52.469+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:52.693 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:52.469+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:52.693 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:52.470+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:53.264 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:52.960+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:53.265 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:52.960+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:53.265 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:52.960+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:53.265 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:52 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:52.960+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:53.553 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:53.262+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:53.554 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:53.262+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:53.554 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:53.262+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:53.554 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:53.263+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:53.554 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:53.485+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:53.554 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:53.485+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:53.555 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:53.485+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:53.555 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:53.485+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:54.242 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:53.984+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:54.243 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:53.984+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:54.243 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:53.984+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:54.243 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:53 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:53.985+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:54.566 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:54.449+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:54.566 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:54.449+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:54.566 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:54.449+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:54.567 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:54.450+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:54.567 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:54.240+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:54.567 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:54.240+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:54.567 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:54.240+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:54.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:54.241+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:54.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:54 smithi046 systemd[1]: Stopping Ceph osd.0 for b1d3a2d0-0449-11ef-bc93-c7b262605968... 2024-04-27T04:45:54.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:54.563+0000 7f6d60ceb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T04:45:54.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:54.563+0000 7f6d60ceb700 -1 osd.0 47 *** Got signal Terminated *** 2024-04-27T04:45:54.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:54.563+0000 7f6d60ceb700 -1 osd.0 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T04:45:55.286 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:54.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:55.286 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:54.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:55.287 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:54.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:55.287 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:54 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:54.978+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:55.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:55.426+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:55.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:55.426+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:55.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:55.426+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:55.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:55.427+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:55.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:55.283+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:55.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:55.283+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:55.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:55.283+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:55.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:55.284+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:56.304 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:55.970+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:56.304 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:55.970+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:56.304 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:55.970+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:56.305 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:55 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:55.970+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:56.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:56.388+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:56.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:56.388+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:56.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:56.388+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:56.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:56.389+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:56.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:56.301+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:56.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:56.301+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:56.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:56.301+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:56.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:56.302+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:57.317 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:56.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:57.317 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:56.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:57.318 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:56.977+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:57.318 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:56 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:56.978+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:57.318 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:57.315+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:57.319 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:57.315+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:57.321 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:57.315+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:57.321 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:57.316+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:57.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:57.431+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:57.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:57.431+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:57.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:57.431+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:57.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:57 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:57.432+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:58.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:58.267+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:58.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:58.267+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:58.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:58.267+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:58.426 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:58.268+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:58.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:58.027+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:58.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:58.027+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:58.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:58.027+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:58.427 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:58.027+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:58.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:58.480+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:58.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:58.480+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:58.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:58.480+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:58.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:58 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:58.481+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:59.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:59.265+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:59.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:59.265+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:59.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:59.265+0000 7f6d59ada700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:38.501794+0000 (oldest deadline 2024-04-27T04:35:58.501794+0000) 2024-04-27T04:45:59.425 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:45:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0[45689]: 2024-04-27T04:45:59.266+0000 7f6d59ada700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14538.0:8055 3.1f 3:fbf6eda1:::1000000408a.00000000:head [write 0~5618 in=5618b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:45:59.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:59.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:59.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:59.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:59.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:59.033+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:45:59.426 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:45:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:45:59.033+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:45:59.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:59.526+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:59.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:59.526+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:59.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:59.526+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:45:59.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:45:59 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:45:59.527+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:00.375 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:46:00 smithi046 podman[82359]: 2024-04-27 04:46:00.081066992 +0000 UTC m=+5.633869566 container died 563c995c0df64dfd495c2f364a122480a0562a129150b8eb039244857f161874 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, RELEASE=reef-5403a02, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-27T04:46:00.376 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:00.043+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:00.376 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:00.043+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:00.376 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:00.043+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:00.376 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:00.044+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:46:00.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:46:00 smithi046 podman[82359]: 2024-04-27 04:46:00.456258556 +0000 UTC m=+6.009061143 container cleanup 563c995c0df64dfd495c2f364a122480a0562a129150b8eb039244857f161874 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=reef-5403a02, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-04-27T04:46:00.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:46:00 smithi046 bash[82359]: ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0 2024-04-27T04:46:00.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:00.540+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:00.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:00.540+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:00.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:00.540+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:00.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:00 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:00.541+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:01.064 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:46:00 smithi046 podman[82379]: 2024-04-27 04:46:00.732939319 +0000 UTC m=+0.649941511 container remove 563c995c0df64dfd495c2f364a122480a0562a129150b8eb039244857f161874 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-5403a02, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-27T04:46:01.352 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:46:01 smithi046 podman[82468]: 2024-04-27 04:46:00.993424238 +0000 UTC m=+0.021703298 image pull 267a473d505df5b518068223eb9431973cc4aab49462dd79c898418ebe6d341a quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec 2024-04-27T04:46:01.352 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:46:01 smithi046 podman[82468]: 2024-04-27 04:46:01.109009179 +0000 UTC m=+0.137288274 container create 36573413d582cdc622b4fe26b34ffc81c717bf8c087acfe8b6065c3aeaa8a9b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=reef-5403a02, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True) 2024-04-27T04:46:01.352 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:01.062+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:01.352 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:01.062+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:01.353 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:01.062+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:01.353 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:01.063+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:46:01.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:46:01 smithi046 podman[82468]: 2024-04-27 04:46:01.373061889 +0000 UTC m=+0.401340953 container init 36573413d582cdc622b4fe26b34ffc81c717bf8c087acfe8b6065c3aeaa8a9b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=reef-5403a02, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T04:46:01.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:46:01 smithi046 podman[82468]: 2024-04-27 04:46:01.376587769 +0000 UTC m=+0.404866837 container start 36573413d582cdc622b4fe26b34ffc81c717bf8c087acfe8b6065c3aeaa8a9b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=reef-5403a02) 2024-04-27T04:46:01.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:46:01 smithi046 podman[82468]: 2024-04-27 04:46:01.478508606 +0000 UTC m=+0.506787669 container attach 36573413d582cdc622b4fe26b34ffc81c717bf8c087acfe8b6065c3aeaa8a9b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=reef-5403a02, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-27T04:46:01.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:01.524+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:01.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:01.524+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:01.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:01.524+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:01.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:01 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:01.525+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:02.022 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:46:01 smithi046 podman[82468]: 2024-04-27 04:46:01.693437577 +0000 UTC m=+0.721716629 container died 36573413d582cdc622b4fe26b34ffc81c717bf8c087acfe8b6065c3aeaa8a9b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=reef-5403a02) 2024-04-27T04:46:02.360 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:02.020+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:02.360 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:02.020+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:02.360 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:02.020+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:02.360 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:02.021+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:46:02.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:02 smithi046 systemd[1]: Stopping Ceph osd.1 for b1d3a2d0-0449-11ef-bc93-c7b262605968... 2024-04-27T04:46:02.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:46:02 smithi046 podman[82503]: 2024-04-27 04:46:02.358008982 +0000 UTC m=+0.659217508 container remove 36573413d582cdc622b4fe26b34ffc81c717bf8c087acfe8b6065c3aeaa8a9b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, RELEASE=reef-5403a02, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424) 2024-04-27T04:46:02.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:46:02 smithi046 systemd[1]: ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-27T04:46:02.675 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:46:02 smithi046 systemd[1]: Stopped Ceph osd.0 for b1d3a2d0-0449-11ef-bc93-c7b262605968. 2024-04-27T04:46:02.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 27 04:46:02 smithi046 systemd[1]: ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@osd.0.service: Consumed 20min 38.405s CPU time. 2024-04-27T04:46:02.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:02.547+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:02.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:02.547+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:02.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:02.547+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:02.676 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:02.548+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:03.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:02.852+0000 7f71ea1fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T04:46:03.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:02.852+0000 7f71ea1fa700 -1 osd.1 47 *** Got signal Terminated *** 2024-04-27T04:46:03.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:02.852+0000 7f71ea1fa700 -1 osd.1 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T04:46:03.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:02.980+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:03.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:02.980+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:03.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:02.980+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:03.176 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:02 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:02.982+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:46:03.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:03.508+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:03.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:03.508+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:03.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:03.508+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:03.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:03.509+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:04.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:03.996+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:04.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:03.996+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:04.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:03.996+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:04.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:03 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:03.996+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:46:04.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:04.469+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:04.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:04.469+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:04.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:04.469+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:04.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:04 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:04.470+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:05.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:05.026+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:05.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:05.026+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:05.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:05.026+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:05.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:05.026+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:46:05.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:05.499+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:05.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:05.499+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:05.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:05.499+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:05.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:05 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:05.499+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:06.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:06.003+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:06.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:06.003+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:06.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:06.003+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:06.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:06.003+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:46:06.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:06.493+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:06.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:06.493+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:06.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:06.493+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:06.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:06.494+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:07.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:06.994+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:07.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:06.994+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:07.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:06.994+0000 7f71e6ff1700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.741420+0000 (oldest deadline 2024-04-27T04:35:54.741420+0000) 2024-04-27T04:46:07.425 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:06 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1[53356]: 2024-04-27T04:46:06.995+0000 7f71e6ff1700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14488.14:831 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=82231b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-27T04:46:07.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:07.461+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:07.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:07.461+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:07.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:07.461+0000 7f7e0202e700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.73:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:35:34.778459+0000 (oldest deadline 2024-04-27T04:35:54.778459+0000) 2024-04-27T04:46:07.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:07 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:07.462+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:08.647 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:08 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:08.438+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:08.648 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:08 smithi046 podman[82582]: 2024-04-27 04:46:08.386994677 +0000 UTC m=+5.658778229 container died 5673c75e53bb5c13b104a13109ca06cefc2c9bae5d7d28916de74f15e9ae8257 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=reef-5403a02, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T04:46:08.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:08 smithi046 podman[82582]: 2024-04-27 04:46:08.736479303 +0000 UTC m=+6.008262855 container cleanup 5673c75e53bb5c13b104a13109ca06cefc2c9bae5d7d28916de74f15e9ae8257 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-5403a02, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T04:46:08.923 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:08 smithi046 bash[82582]: ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1 2024-04-27T04:46:09.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:08 smithi046 podman[82600]: 2024-04-27 04:46:08.921280182 +0000 UTC m=+0.525663076 container remove 5673c75e53bb5c13b104a13109ca06cefc2c9bae5d7d28916de74f15e9ae8257 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1, RELEASE=reef-5403a02, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-27T04:46:09.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:09 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:09.464+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:09.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:09 smithi046 podman[82687]: 2024-04-27 04:46:09.256473958 +0000 UTC m=+0.018664556 image pull 267a473d505df5b518068223eb9431973cc4aab49462dd79c898418ebe6d341a quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec 2024-04-27T04:46:09.675 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:09 smithi046 podman[82687]: 2024-04-27 04:46:09.372220372 +0000 UTC m=+0.134411000 container create d392f1b3e58e02bf55e9df0a3cfa09a877fb06d0e2186ecc28f63bc48c7ee3ad (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-5403a02, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-27T04:46:09.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:09 smithi046 podman[82687]: 2024-04-27 04:46:09.599552726 +0000 UTC m=+0.361743313 container init d392f1b3e58e02bf55e9df0a3cfa09a877fb06d0e2186ecc28f63bc48c7ee3ad (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=reef-5403a02, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-27T04:46:09.676 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:09 smithi046 podman[82687]: 2024-04-27 04:46:09.60283796 +0000 UTC m=+0.365028549 container start d392f1b3e58e02bf55e9df0a3cfa09a877fb06d0e2186ecc28f63bc48c7ee3ad (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=reef-5403a02, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-27T04:46:10.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:09 smithi046 podman[82687]: 2024-04-27 04:46:09.674362083 +0000 UTC m=+0.436552678 container attach d392f1b3e58e02bf55e9df0a3cfa09a877fb06d0e2186ecc28f63bc48c7ee3ad (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=reef-5403a02, io.buildah.version=1.33.5) 2024-04-27T04:46:10.175 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:09 smithi046 podman[82687]: 2024-04-27 04:46:09.907514667 +0000 UTC m=+0.669705253 container died d392f1b3e58e02bf55e9df0a3cfa09a877fb06d0e2186ecc28f63bc48c7ee3ad (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.vendor=CentOS, RELEASE=reef-5403a02, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-27T04:46:10.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:10 smithi046 podman[82713]: 2024-04-27 04:46:10.452915209 +0000 UTC m=+0.540687974 container remove d392f1b3e58e02bf55e9df0a3cfa09a877fb06d0e2186ecc28f63bc48c7ee3ad (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=reef-5403a02) 2024-04-27T04:46:10.925 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:10 smithi046 systemd[1]: ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-27T04:46:10.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:10 smithi046 systemd[1]: Stopped Ceph osd.1 for b1d3a2d0-0449-11ef-bc93-c7b262605968. 2024-04-27T04:46:10.926 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 27 04:46:10 smithi046 systemd[1]: ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@osd.1.service: Consumed 21min 55.547s CPU time. 2024-04-27T04:46:10.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:10.497+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:10.926 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:10 smithi046 systemd[1]: Stopping Ceph osd.2 for b1d3a2d0-0449-11ef-bc93-c7b262605968... 2024-04-27T04:46:11.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:10.930+0000 7f7e05237700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T04:46:11.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:10.930+0000 7f7e05237700 -1 osd.2 47 *** Got signal Terminated *** 2024-04-27T04:46:11.425 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:10 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:10.930+0000 7f7e05237700 -1 osd.2 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T04:46:11.853 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:11 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:11.456+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:12.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:12 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:12.468+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:13.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:13 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:13.453+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:14.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:14 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:14.429+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:15.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:15 smithi046 ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2[61210]: 2024-04-27T04:46:15.469+0000 7f7e0202e700 -1 osd.2 47 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.14538.0:8050 3.7 3:e1d1645b:::100000044e2.00000000:head [write 0~5710 in=5710b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-27T04:46:16.551 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:16 smithi046 podman[82793]: 2024-04-27 04:46:16.297152644 +0000 UTC m=+5.478011849 container died 9787b5012842821ed13598fb08f8016de1e9cfedb15c823787da4e1983eaffac (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=reef-5403a02, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-27T04:46:16.851 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:16 smithi046 podman[82793]: 2024-04-27 04:46:16.631374363 +0000 UTC m=+5.812233576 container cleanup 9787b5012842821ed13598fb08f8016de1e9cfedb15c823787da4e1983eaffac (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2, RELEASE=reef-5403a02, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-27T04:46:16.851 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:16 smithi046 bash[82793]: ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2 2024-04-27T04:46:17.175 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:16 smithi046 podman[82813]: 2024-04-27 04:46:16.849250018 +0000 UTC m=+0.544362842 container remove 9787b5012842821ed13598fb08f8016de1e9cfedb15c823787da4e1983eaffac (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=reef-5403a02, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T04:46:17.612 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:17 smithi046 podman[82904]: 2024-04-27 04:46:17.175164212 +0000 UTC m=+0.018735669 image pull 267a473d505df5b518068223eb9431973cc4aab49462dd79c898418ebe6d341a quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec 2024-04-27T04:46:17.613 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:17 smithi046 podman[82904]: 2024-04-27 04:46:17.283510306 +0000 UTC m=+0.127081742 container create 007938e45c77c6e405a31bd0c253d5967279e8b791338280e072f5f976681460 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=reef-5403a02, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-04-27T04:46:17.613 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:17 smithi046 podman[82904]: 2024-04-27 04:46:17.519078535 +0000 UTC m=+0.362649965 container init 007938e45c77c6e405a31bd0c253d5967279e8b791338280e072f5f976681460 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=reef-5403a02, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-27T04:46:17.613 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:17 smithi046 podman[82904]: 2024-04-27 04:46:17.522333291 +0000 UTC m=+0.365904720 container start 007938e45c77c6e405a31bd0c253d5967279e8b791338280e072f5f976681460 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=reef-5403a02, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-04-27T04:46:17.613 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:17 smithi046 podman[82904]: 2024-04-27 04:46:17.610696136 +0000 UTC m=+0.454267569 container attach 007938e45c77c6e405a31bd0c253d5967279e8b791338280e072f5f976681460 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=reef-5403a02, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-27T04:46:17.925 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:17 smithi046 podman[82904]: 2024-04-27 04:46:17.820473953 +0000 UTC m=+0.664045407 container died 007938e45c77c6e405a31bd0c253d5967279e8b791338280e072f5f976681460 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=reef-5403a02, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T04:46:18.365 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:18 smithi046 podman[82930]: 2024-04-27 04:46:18.339330433 +0000 UTC m=+0.514106297 container remove 007938e45c77c6e405a31bd0c253d5967279e8b791338280e072f5f976681460 (image=quay.ceph.io/ceph-ci/ceph@sha256:f9a5d4aaa8713f713697a22420a6f314a98f32eda5ca1cf607bd68f86325baec, name=ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=reef-5403a02, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-27T04:46:18.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:18 smithi046 systemd[1]: ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-27T04:46:18.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:18 smithi046 systemd[1]: Stopped Ceph osd.2 for b1d3a2d0-0449-11ef-bc93-c7b262605968. 2024-04-27T04:46:18.675 INFO:journalctl@ceph.osd.2.smithi046.stdout:Apr 27 04:46:18 smithi046 systemd[1]: ceph-b1d3a2d0-0449-11ef-bc93-c7b262605968@osd.2.service: Consumed 27.065s CPU time. 2024-04-27T04:46:20.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:46:20.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:46:23.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:46:32.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:46:32.520 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:46:35.582 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:46:47.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:46:47.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:46:50.658 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:46:53.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:46:53.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:46:53.726 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:46:53.726 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:46:53.727 INFO:tasks.cephadm:Teardown complete 2024-04-27T04:46:53.727 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephadm.py", line 1706, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephadm.py", line 2246, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephadm.py", line 2249, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephadm.py", line 1839, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephadm.py", line 226, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephadm.py", line 395, in _rm_cluster remote.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi073 2024-04-27T04:46:53.752 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-27T04:46:53.763 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-04-27T04:46:53.780 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-27T04:46:53.780 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-27T04:46:53.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:46:53.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:46:56.862 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:47:05.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:47:05.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:47:08.926 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:47:20.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:47:20.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:47:23.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:47:27.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:47:27.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:47:27.074 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:47:27.074 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:47:27.119 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi073 2024-04-27T04:47:27.139 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-27T04:47:27.154 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-27T04:47:27.154 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T04:47:27.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:47:27.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:47:27.179 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-04-27T04:47:27.183 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T04:47:27.183 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-04-27T04:47:27.183 INFO:teuthology.orchestra.run.smithi046.stdout:^* hv01.front.sepia.ceph.com 3 6 377 25 +684us[ +649us] +/- 61ms 2024-04-27T04:47:27.184 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 89 -1779us[-1818us] +/- 62ms 2024-04-27T04:47:27.184 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +1523us[+1523us] +/- 71ms 2024-04-27T04:47:27.184 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T04:47:30.238 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:47:39.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:47:39.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:47:42.302 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:47:54.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:47:54.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:47:57.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:48:00.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:48:00.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:48:00.446 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:48:00.446 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:48:00.447 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi073 2024-04-27T04:48:00.461 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-27T04:48:00.475 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-27T04:48:00.475 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-27T04:48:00.494 DEBUG:teuthology.orchestra.run.smithi046:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-27T04:48:00.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:48:00.581 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:48:03.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:48:12.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:48:12.648 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:48:15.710 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:48:27.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:48:27.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:48:30.782 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:48:33.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:48:33.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:48:33.854 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:48:33.854 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:48:33.855 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi073 2024-04-27T04:48:33.864 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-27T04:48:33.876 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-27T04:48:33.888 INFO:teuthology.task.internal:Duration was 3671.665029 seconds 2024-04-27T04:48:33.888 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-27T04:48:33.901 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-27T04:48:33.901 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T04:48:33.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:48:33.908 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:48:33.951 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T04:48:36.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:48:45.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:48:45.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:48:49.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:49:01.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:49:01.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:49:04.094 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:49:07.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:49:07.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:49:07.170 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:49:07.170 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:49:07.171 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi073 2024-04-27T04:49:07.187 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-27T04:49:07.199 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-27T04:49:07.199 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T04:49:07.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:49:07.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:49:10.306 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:49:19.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:49:19.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:49:22.366 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:49:34.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:49:34.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:49:37.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:49:40.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:49:40.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:49:40.514 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:49:40.514 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:49:40.515 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi073 2024-04-27T04:49:40.533 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-27T04:49:40.547 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T04:49:40.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:49:40.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:49:40.580 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2024-04-27T04:49:43.614 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:49:52.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:49:52.616 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:49:55.678 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:50:07.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:50:07.681 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:50:10.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:50:13.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:50:13.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:50:13.825 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:50:13.826 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:50:13.827 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi073 2024-04-27T04:50:13.843 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-27T04:50:13.856 INFO:teuthology.task.internal:Transferring archived files... 2024-04-27T04:50:13.857 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-27_03:08:09-upgrade-main-distro-default-smithi/7675777/remote/smithi046 2024-04-27T04:50:13.857 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T04:50:13.898 DEBUG:teuthology.misc:Transferring archived files from smithi073:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-27_03:08:09-upgrade-main-distro-default-smithi/7675777/remote/smithi073 2024-04-27T04:50:13.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:50:13.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:50:16.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:50:25.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:50:25.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:50:29.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:50:41.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:50:41.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:50:44.094 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:50:47.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:50:47.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:50:47.170 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:50:47.170 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:50:47.171 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi073 2024-04-27T04:50:47.188 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-27T04:50:47.201 INFO:teuthology.task.internal:Not uploading archives. 2024-04-27T04:50:47.201 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-27T04:50:47.212 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-27T04:50:47.213 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T04:50:47.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:50:47.220 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:50:47.236 INFO:teuthology.orchestra.run.smithi046.stdout: 262213 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 27 04:46 /home/ubuntu/cephtest 2024-04-27T04:50:47.237 INFO:teuthology.orchestra.run.smithi046.stdout: 263988 212 -rwxr-xr-x 1 ubuntu ubuntu 215333 Apr 27 03:53 /home/ubuntu/cephtest/cephadm 2024-04-27T04:50:47.237 INFO:teuthology.orchestra.run.smithi046.stdout: 262214 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Apr 27 04:49 /home/ubuntu/cephtest/archive 2024-04-27T04:50:47.237 INFO:teuthology.orchestra.run.smithi046.stdout: 675956 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 04:48 /home/ubuntu/cephtest/archive/audit 2024-04-27T04:50:47.237 INFO:teuthology.orchestra.run.smithi046.stdout: 675958 180 -rw------- 1 ubuntu ubuntu 180651 Apr 27 04:48 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-04-27T04:50:47.237 INFO:teuthology.orchestra.run.smithi046.stdout: 262216 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 03:47 /home/ubuntu/cephtest/archive/syslog 2024-04-27T04:50:47.237 INFO:teuthology.orchestra.run.smithi046.stdout: 262218 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Apr 27 03:47 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T04:50:47.238 INFO:teuthology.orchestra.run.smithi046.stdout: 262217 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Apr 27 03:47 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T04:50:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-27T04:50:50.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:50:59.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:50:59.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:51:02.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:51:14.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:51:14.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:51:17.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:51:20.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-04-27T04:51:20.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:51:20.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-04-27T04:51:20.478 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:51:20.479 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_3067e9f133f954b642fdf21114a79edb58590948/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi073 2024-04-27T04:51:20.496 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-27T04:51:20.512 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-27T04:51:20.531 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-04-27T04:51:20.531 INFO:teuthology.nuke:Checking targets against current locks 2024-04-27T04:51:20.562 DEBUG:teuthology.nuke:shortname: smithi046 2024-04-27T04:51:20.562 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T04:51:20.589 DEBUG:teuthology.nuke:shortname: smithi073 2024-04-27T04:51:20.590 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T04:51:20.594 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-27_03:08:09-upgrade-main-distro-default-smithi/7675777', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 03:39:02.274705', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN8oH4piT6dsshEaKEHjPhB9qziQxQLIzhIPaH6npcdj5tUJpND+yC26uLg+lUe8/WUJwV+YV1lDUHFKbAfbRG8='} 2024-04-27T04:51:20.619 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi073.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-27_03:08:09-upgrade-main-distro-default-smithi/7675777', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 03:39:02.276077', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPOUmIHJ1G/Dby/RkB4+wbyo3AYAEeb02AiGYv/PoYzQzexIeTL4B4X8pqVMHLUxJNDWSKdCHC3xutGxKrdGNpk='} 2024-04-27T04:51:20.647 INFO:teuthology.orchestra.console.smithi046:Power off 2024-04-27T04:51:20.647 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-27T04:51:20.676 DEBUG:teuthology.orchestra.console.smithi046:power off output: Chassis Power Control: Down/Off 2024-04-27T04:51:20.676 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T04:51:20.694 INFO:teuthology.orchestra.console.smithi073:Power off 2024-04-27T04:51:20.695 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-27T04:51:20.707 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-04-27T04:51:20.720 DEBUG:teuthology.orchestra.console.smithi073:power off output: Chassis Power Control: Down/Off 2024-04-27T04:51:20.720 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T04:51:20.744 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2024-04-27T04:51:24.709 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T04:51:24.746 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T04:51:24.820 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-04-27T04:51:24.857 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2024-04-27T04:51:28.822 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T04:51:28.858 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T04:51:28.935 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is off 2024-04-27T04:51:28.968 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2024-04-27T04:51:29.037 INFO:teuthology.orchestra.console.smithi046:Power off completed 2024-04-27T04:51:32.972 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T04:51:33.083 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is off 2024-04-27T04:51:33.184 INFO:teuthology.orchestra.console.smithi073:Power off completed 2024-04-27T04:51:33.289 INFO:teuthology.run:Summary data: description: upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3671.6650292873383 failure_reason: 'Command failed on smithi046 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -e sha1=4e213a58e787437ff508e6c890ccd5fd32dc96b5 -- bash -c ''ceph orch ps''"' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ede868120d52474badaab00dc4a2e71a status: fail success: false 2024-04-27T04:51:33.289 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-27T04:51:33.380 INFO:teuthology.run:FAIL